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Progress, far from consisting in change, depends on retentiveness. 
When change is absolute there remains no being to improve and no 
direction is set for possible improvement: and when experience is 
not retained, as among savages, infancy is perpetual. Those who 
cannot remember the past are condemned to repeat it. In the first 
stage of life the mind is frivolous and easily distracted, it misses 
progress by failing in consecutiveness and persistence. This is the 
condition of children and barbarians, in which instinct has learned 
nothing from experience. 

George Santayana, The Life of Reason, Volume 1, 1905 
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1.0 Introduction 


This report is the final product of a 90 day study performed for the Exploration Systems Mission 
Directorate. The study was to assemble lessons NASA has learned from previous programs that 
could help the Exploration Systems Mission Directorate pursue the Exploration vision. It 
focuses on those lessons that should have the greatest significance to the Directorate during the 
formulation of program and mission plans. 

The study team reviewed a large number of lessons learned reports and data bases, including the 
Columbia Accident Investigation Board and Rogers Commission reports on the Shuttle 
accidents, accident reports from robotic space flight systems, and a number of management 
reviews by the Defense Sciences Board, Government Accountability Office, and others. The 
consistency of the lessons, findings, and recommendations validate the adequacy of the data set. 

In addition to reviewing existing databases, a series of workshops was held at each of the NASA 
centers and headquarters that included senior managers from the current workforce as well as 
retirees. The full text of the workshop reports is included in Appendix A. A lessons learned 
website was opened up to permit current and retired NASA personnel and on-site contractors to 
input additional lessons as they arise. These new lessons, when of appropriate quality and 
relevance, will be brought to the attention of managers. 

The report consists of four parts: 

Part 1 provides a small set of lessons, called the Executive Lessons Learned, that represent 
critical lessons that the Exploration Systems Mission Directorate should act on immediately. 

This set of Executive Lessons and their supporting rationale have been reviewed at length and 
fully endorsed by a team of distinguished NASA alumni. 

Part 2 contains a larger set of lessons, called the Selected Lessons Learned, which have been 
chosen from the lessons database and center workshop reports on the basis of their specific 
significance and relevance to the near-term work of the Exploration Directorate. These lessons 
frequently support the Executive lessons but are more general in nature. 

Part 3 consists of the reports of the center workshops that were conducted as part of this activity. 
These reports are included in their entirety (approximately 200 pages) in Appendix G and have 
significance for specific managers. 

Part 4 consists of the remainder of the lessons that have been selected by this effort and 
assembled into a database for the use of the Explorations Directorate. The database is archived 
and hosted in the Lessons Learned Knowledge Network, which provides a flexible search 
capability using a wide variety of search terms. 
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Finally, a spreadsheet lists databases searched and a bibliography identifies reports that have 
been reviewed as sources of lessons for this task. 

NASA has been presented with many learning opportunities. We have conducted numerous 
programs, some extremely successful and others total failures. Most have been documented with 
a formal lessons learned activity, but we have not always incorporated these learning 
opportunities into our normal modes of business. For example, the Robbins Report of 2001 
clearly indicates that many project failures of the past two decades were the result of violating 
well documented best practices, often in direct violation of management instructions and 
directives. An overarching lesson emerges: that disciplined execution in accordance with proven 
best practices is the greatest single contributor to a successful program. 

The Lessons Learned task team offers a sincere hope that the lessons presented herein will be 
helpful to the Exploration Systems Directorate in charting and executing their course. The 
success of the Directorate and of NASA in general depends on our collective ability to move 
forward without having to relearn the lessons of those who have gone before. 
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2.0 Executive Summary 


The Exploration Systems Mission Directorate leads the most challenging adventure ever given a 
team of humans: develop and implement plans for the human exploration of the moon and planet 
Mars. This program follows in the footsteps of numerous other human space efforts over the 
past 40 years. It can and must build on the successes of those past programs and avoid their 
mistakes. A cost-effective implementation of the Exploration Vision would be impossible if the 
implementing organizations needed to develop critical skills and knowledge on their own by trial 
and error 

Soon after beginning this effort, the team uncovered an enormous number of documented 
lessons. This very large set of lessons was screened for those that are appropriate for 
consideration by the Exploration Directorate, and this smaller set was sorted into three groups. 

The first group consists of a small set of critical lessons, called the Executive Lessons, which 
should be embraced by the Exploration leadership immediately. They include the first-principles 
that set human space flight apart from other endeavors, as well as the attitudes and top level 
approaches that determine the success or failure of the effort. These lessons have been 
developed by synthesizing lessons, interviews, and workshop discussions, and represent the 
collective wisdom of a broad group of experts. They have been accepted by all reviewers and 
endorsed for their overwhelming importance to program success. 

A second group of lessons, called the Selected Lessons, is important for the general process of 
managing the effort. These lessons can be of value to many levels of the organization, but still 
have their greatest impact on the top level management of major technical programs and 
especially human space flight programs. 

The third group is a much larger set of lessons that applies to the lower level technical and 
operational personnel in the performance of their jobs. These are to be found in the reports of 
the center workshops, the database provided with this report, and the documents identified in the 
bibliography. 

Executive Lessons Learned: 

There are seven Executive Lessons Learned, each of which is developed and explained in detail: 

1 . Human Space is Different: The inherent risk and extreme design optimization required 
for human spacecraft and missions makes their technical design and operations 
techniques different from those of other technical programs. 

2. Focus on Mission Success: The highly demanding technical and operational 
requirements of human space missions require keeping mission success as the principle 
management focus. Manage the quality and reliability of the system, and the cost 
performance will follow. 
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3. Understand and reinforce the roles of the government and the contractor: The 

highly specialized needs of human space flight coupled with the long intervals between 
development programs create a situation that requires strategic management and 
cooperation between the government and contractors. 

4. Human space flight experience is critical: The experience of human space flight 
developers, managers, and operators is different from that of other engineering 
endeavors. In order to avoid repeating costly mistakes, key leaders in these areas must 
have personal experience in the unique opportunities and traps of human space flight. 

5. Under-funded programs are invariably flawed: Programs that are significantly under 
funded at initiation place an almost irresistible pressure on the program manager to take 
technical and operational shortcuts that add substantial risk to the program. 

6. Early identification of appropriate, validated requirements is key: The complexity 
of human space systems makes their success totally dependent on effective systems 
engineering. Requirements identification and management is the life-blood of systems 
engineering, so the success of the program depends critically on the quality of the 
requirements provided at program implementation. 

7. Plan for the impact of failures: The Shuttle has had two catastrophic accidents, both of 
which have had major impacts on the future of space flight. Any major, long-tenn 
human space flight program can expect to have occasional, unexpected failures, so the 
program must be designed to accommodate those failures without excessive disruption to 
other program activities. 

Appendix A contains 85 Selected Lessons grouped into categories such as Program 
Management, Systems Engineering, Design Principles, and so on. These represent those that 
have been found in documentation and presented as personal experience during the Center 
Workshops. 

Appendix B lists the NASA Center workshops that have been conducted in support of this 
activity. The reports of these workshops can be found on the CD of supporting data. 

Appendix C contains the bibliography of sources from which the study team drew the lessons 
learned. This list is not complete, as new potential sources were being provided to the team up to 
the time this report was published 

Appendix D contains a description of the process used in this lessons learned activity, including 
the points of contact at the Centers who conducted the highly successful workshops. 

Appendix E contains selected citations to support the Executive Lessons. 

Appendix F contains a recommendation for Phase 2 Lessons Learned effort. 

Appendix G (CD version only) provides an electronic version of the workshop reports, LLKN 
database, and other supporting infonnation. 
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3.0 Results — Executive Lessons 


Seven lessons have been selected as the highest priority for a major new program such as the 
Exploration Vision. They represent what this team believes are the most important lessons 
learned in space development and operations. If followed, these lessons should contribute 
significantly to the success of the program. They emerge from numerous sources, including 
hundreds of lessons learned reviewed by the team, the Rogers Commission and Columbia 
Accident Investigation Board, DOD reports, Government Accountability Office reports, and 
lessons learned workshops conducted for this effort at the centers. 

NASA was created to pursue space exploration for the United States. The first task of the new 
organization was to identify the technical capabilities needed to be successful in this endeavor 
and assemble them to enable the new enterprise. The core of the new capability was found in the 
technical laboratories that made up the NACA and the Redstone Arsenal, while the program 
management talent was drawn from industry and the department of defense. Together they 
formed the nucleus of a technical and operational capability that delivered humans to the moon 
in barely a decade, and that has continued to be the preeminent human space flight organization 
in the world. 

Much of that capability was lost in the years since Apollo because there were so few human 
space system development programs. The remaining capability is largely devoted to supporting 
the safe operation of the space flight programs in existence today. Several of the lessons in this 
package deal with the significance of this technical capability and how it can be harnessed to 
facilitate the success of the new Exploration mission. 

These seven lessons have been reviewed by a distinguished panel of NASA alumni and received 
a strong endorsement as critical to the success of the new program. The discussions 
accompanying the lessons provide understanding of the context from which the lessons are 
drawn, and the recommendations offer action statements for implementation. Appendix E 
contains a list of citations of source material that can provide further information on the origins 
of the lessons. 


1. Lesson: Human space is different. 

A human space flight program is different than other development and operational 
activities. It is inherently much more critical and stressing to all technologies and 
competencies involved and requires the highest levels of engineering and management 
performance to achieve even minimum levels of mission success. (Defense Sciences 
Board, multiple sources) 

Discussion: 
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The space industry grew out of the aircraft business and initially applied the same 
engineering and operational techniques to this new field. As documented extensively in 
the lessons, the extremely high failure rate of the early space systems demonstrated that 
existing techniques did not accommodate the increased system stresses, performance 
uncertainties, and consequences of failure found in space systems. For example, a loss of 
electrical power in an aircraft may still permit it to land safely, but a similar failure in a 
space system will result in complete failure. Almost all engineering, design, and 
operational protocols were challenged beyond anything previously attempted. This 
lesson was learned through many spectacular failures in those early programs. The path 
to acceptable design and operational techniques has been long and expensive, and is not 
yet complete. 

Space designs are sensitive to parameters other endeavors consider inconsequential. For 
example, Apollo 15 almost aborted the Lunar landing due to a solder ball floating up to 
short the tenninals of an abort switch in the LM. On Shuttle, the difference in activating 
pushbutton switches in zero G versus on Earth has been shown to sometimes render the 
switch inoperative. This study found many examples where EMI, contaminant control, 
electrical grounding, and many other disciplines felt to be well controlled in terrestrial 
and aviation fields become sources of new failure modes for space systems. 

Human space flight is an extremely hazardous activity. From Mercury to STS- 107, the 
US has demonstrated an overall catastrophic failure rate for human space missions of 1 
failure per 72 missions, or the loss of one life per 8 missions. In addition, there have 
been numerous close calls that could easily have resulted in further losses. 

Despite these statistics, NASA has become highly proficient at many of the elements of 
space systems design, manufacture, and operations. When well-tested processes are 
followed rigorously, the success rate has been good. When these processes have been 
shortcut as a result of cost or schedule pressure, errors and omissions have crept in. Both 
of the Shuttle accidents had their origins in elements of the spacecraft design that were 
not as well understood as they should have been, but in both cases there were 
opportunities for established processes that are unique to the high-risk human space flight 
industry to catch them if they had been managed properly. Both accidents followed 
periods of increased political and budget pressure that contributed to relatively minor 
technical issues becoming national tragedies. 

While NASA has become fairly proficient in operations in low Earth orbit, it remains a 
novice at human exploration missions. In terms of the personal experience and corporate 
memory available for the new mission, NASA is at the level of maturity equivalent to the 
Mercury and Gemini missions relative to Apollo. This means that incremental 
performance enhancements coupled with extremely high levels of analysis and testing 
will be required to keep the risk of catastrophic mission failure to acceptable levels. 
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Because space missions rarely permit rapid return following failures and offer few 
opportunities for safe abort or bailout, safety must be achieved through extreme attention 
to reliability and knowledge of system state. This requires highly reliable systems that 
are thoroughly modeled, analyzed, tested, instrumented, and constantly monitored by a 
large team of experts to detect the earliest signs of incipient failure. The engineering and 
design techniques to achieve this level of perfonnance are not common elsewhere, and 
require unique design approaches. Testing and failure analysis techniques are likewise 
elevated to the highest possible levels. 

System knowledge and training for flight controllers, engineering support personnel, and 
crew are exceptionally high. Aborts and other recovery responses to failures are 
extensively pre-planned and trained. Some test engineers have observed that an aircraft 
flight test can open the flight envelope incrementally, while spacecraft flight test 
generally utilizes the entire flight envelope on the first flight. 

Space is a “one strike and you are out” business. Any significant error or flaw can turn a 
multi-billion dollar mission into a catastrophe. Many failure modes are catastrophic, with 
little warning or opportunity to respond. For launch vehicles, even a minor failure that 
does not result in an explosion can reduce performance enough to cause total mission 
failure. 

It has been possible to design this level of perfonnance criticality out of commercial air 
carrier aircraft because the relatively benign thrust to weight ratios reduce system and 
performance stresses. For example, air carrier aircraft are designed and operated to 
survive the loss of an engine anywhere during the mission. The loss of an engine in a 
launch vehicle will usually result in a mission failure. 

Extremely high levels of reliability and knowledge are required for success across the 
mission profile. Increasing the confidence in the safety of the vehicle and crew enables 
the mission manager to continue the mission in the presence of various combinations of 
failures that otherwise could result in an abort or early mission tennination. 

The Exploration mission will be even more difficult and challenging than what has 
preceded it. The long durations of Exploration missions, coupled with the fewer and 
more tenuous opportunities for abort and/or rescue, increase the risk by an order of 
magnitude or more. Addressing this risk will require an unprecedented level of reliability 
in all of the critical vehicle systems. Although reliability requirements for nuclear 
systems approach those that will be needed for the Exploration mission, spacecraft cannot 
afford the weight penalties that nuclear systems employ to achieve their huge safety 
factors. Thus, Exploration will require major breakthroughs in lightweight, reliable 
systems technology and design. 

In addition, the Exploration mission will never achieve a routine operational capability. 
All mission preparations, including hardware/software design and test, as well as 
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operations, will continue to require processes and protocols reflecting the demands and 
rigor of experimental flight test. 

The sum of this lesson is that human space is far more difficult and dangerous than either 
robotic space or modern aviation. The path to mission success is found by pursuing the 
highest possible system reliability through all aspects of system design, manufacturing, 
test, and operations. There is no precedent in any other technical endeavor for this level 
of performance and reliability. 

Recommendation : 

The program manager must ensure that the highest quality processes are in place for all 
aspects of the program. The management approach, thought processes, and decision 
criteria are different from other technology development and test activities. An 
unprecedented focus on highly reliable systems is mandatory. 

2. Lesson: Focus on mission success 

Excessive attention to cost and schedule performance has a negative impact on mission 
success. Mission success in space systems is most often achieved when it is the primary 
focus of the organization and the management processes that support it. 

Discussion: 

Much of Apollo’s extraordinary success is attributed to its virtually unlimited funding. 
The program’s technical and management teams spent relatively little time worrying 
about cost, focusing instead on developing the hardware and software needed for mission 
success on schedule. NASA today does not have the same political climate and 
budgetary support, and program personnel are expected to be highly cognizant of the 
budget for their work and ensure that program milestones are met. 

Giving equal priority to cost and schedule relative to mission success can be very 
damaging to the program. Studies of management initiatives such as “Faster, Better, 
Cheaper” demonstrate that such a focus significantly reduces the probability of mission 
success. For example, the Robins report of 2001 found that while “FBC” implementation 
the 1990’s did reduce the cost and schedule of the projects, the success rate of the 
missions needed improvement. “Faster, Better, Cheaper” was only one attempt to treat 
space systems like other technology endeavors, with cost and schedule performance equal 
to or superior to technical perfonnance, and the increased failure rate and cost overruns it 
created demonstrate the ineffectiveness of this approach. The respected authority Tom 
Young says succinctly, “In the last decade. . .the primary focus of space programs has 
been cost. You can’t achieve cost performance by managing cost. You achieve it by 
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managing reliability. What would a fraction of the $1 IB in losses over the past decade 
have bought in terms of mission success in the 1990s?” 

Numerous lessons in the database point to the requirement to maintain focus on mission 
success as the primary metric for managing performance and predicting program success. 
Insisting on quality in the design effort results in systems that can be successfully built, 
tested, integrated, and flown at minimum cost. Attempts to save money and time by 
skipping elements of traditional engineering and management effort lead to expensive 
and time consuming errors in requirements and system design, test failures, and redesign 
and regression testing. 

Mission success requires top quality attention to all aspects of the program, including 
requirements definition and all aspects of management planning. When the requirements, 
WBS, and other planning are inadequate, issues that are critical to mission success are 
lost. This causes cost and schedule to compete with critical technical performance as 
issues of the day and distract the management team from those issues that have the 
greatest impact on mission success. Shortcuts taken in the requirements phase show up 
as major problems in the design phase. Shortcuts in design show up in integrated test or 
operations, with far greater impact to the program. But when resources are applied up 
front to ensure that the system is being designed correctly and work is being done on 
time, the cost of the planning and management effort is repaid many times in savings for 
hardware/software rework and regression testing. 

The high cost of space systems sometimes results in management cutting back on 
technical and program management tools and resources. The consequences of reduced 
investment in management are generally negative. Numerous innovative approaches 
have tried to reduce the cost of engineering and managing space programs, but very few 
such innovations have succeeded in the long run. For example, the International Space 
Station Program attempted to substantially reduce the requirements for business 
management reporting only to find the lack of timely reporting delayed identification of 
and response to significant performance issues. Likewise, several key technical 
management products, such as integrated schematics, were either significantly down- 
sized or eliminated outright. Virtually all of these documents had to be added back at a 
higher net cost to the program. 

Experience here in the US and abroad clearly demonstrates that the surest path to mission 
success is quality of requirements and design, including a robust system that can tolerate 
errors and failures. The Russians in particular have demonstrated the enormous 
importance of system robustness and resilience, particularly in periods of resource 
scarcity. 

Recommendation: 
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Program managers must clearly establish management processes that focus on those 
elements of the program that most strongly influence mission success. These include 
thorough program planning and a robust and resilient system design. This requires 
pursuing the highest achievable quality in ah program activities, technical and 
management. Manage the quality and reliability of the system in the engineering phase, 
and the cost performance will follow. 


3. Lesson: Understand and reinforce the roles of the government and 
contractor teams 

The appropriate balance of management and technical capabilities must be maintained 
between the NASA and contractor communities. The US aerospace contractors cannot 
execute a major human space flight program without substantial NASA technical 
participation, and NASA does not have the workforce to engineer and manufacture a 
major flight system. Therefore, managers must maintain the correct balance of capability 
and responsibility between the communities, including programs to guarantee the supply 
of future technical and management experts. 

Discussion: 

New space flight programs are not common, and new human space flight programs are 
extremely rare. Industry can only afford to maintain its workforce and infrastructure to 
support its current business; it cannot maintain a skilled human space flight workforce 
between programs. This means that industry cannot be counted on to have the trained 
and experienced workforce needed to execute a major human space program. For each 
new program, industry will need to develop a large fraction of the skills that will be 
required, a process that can take years and cost many millions of dollars. This knowledge 
and experience is only available in the NASA subsystem managers and engineers who 
have overseen the design, development, and operation of previous generations of space 
flight systems. 

What contractors bring to the table is the ability to generate the large workforce required 
to produce the detailed design and manufacture the hardware. NASA cannot support the 
infrastructure to perform those tasks during the long periods between flight programs. 
Therefore, the capabilities of the two groups are synergistic and must be managed in 
concert to ensure an efficient and timely development program. NASA retains the long- 
term technical expertise, and the contractors have the flexible workforce and 
manufacturing capability. 

During the proposal stage, the contractors will come to NASA for this expertise to 
support proposal preparation and design work. They will continue to depend on 
extensive dialogs with NASA subsystem experts as the designs mature through 
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development, test, manufacture, and operations. The contractor then produces a detailed 
design and manufactures the hardware under the watchful eye of the NASA experts, who 
also support the integration, test and operations phases. If the NASA workforce were 
unable to provide those capabilities on demand, the contractor would have to develop the 
experience and competence on their own by trial and error, at great expense. 

Therefore, NASA needs to possess sufficient in-house technical resources to participate 
with the contractor teams to address those critical technical needs. This certainly includes 
the traditional role of “smart buyer,” but also includes providing historical and integrated 
technical expertise. 

Contractors must own the final design of the system they produce and be prepared to 
integrate it and test it to prove that it meets all requirements. The contractor must be able 
to support sustaining engineering roles for the life of the system including providing 
detailed anomaly resolution and on-demand engineering support for integrated system 
issues. While these roles are partially shared with government subsystem managers, 
particularly in integration across multiple elements, the role of the contractor is central. 
The detailed understanding of the exact hardware configuration must belong to the 
contractor who developed it. 

Much of the operations phase will be directly managed and performed by government 
employees because the extremely high risk of those phases requires government 
personnel manage and accept that risk. Contractors are not allowed to accept risk on 
behalf of the government beyond very specific limits of traditional design practices. 

Even then, the contractor must advise the government of risks that are known to exist in 
the design and operational elements, and the government management team must accept, 
integrate, and manage those risks. 

Finally, should international partners be part of the program, only civil servants can make 
and modify govemment-to-govemment agreements and bear the risks of non- 
performance by international partners. Contractors may subcontract some work to 
international companies in traditional contract relationships, but govemment-to- 
govemment agreements are outside the scope of their responsibilities. 

NASA needs to support the development and maintenance of this in-house capability. 
Doing so involves ensuring that there are sufficient qualified engineers in the NASA 
workforce and that they stay current in their technical specialties, including emerging 
technologies and management techniques. Center directors should ensure that the civil 
service workforce can meet the demands of current and planned programs. The 
Exploration Systems Directorate should work together with the centers to identify the 
level of capability needed to maintain the core NASA capability and to ensure that 
sufficient investment is made in this capability. 

Recommendation: 
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NASA should develop a robust, comprehensive, and cost effective manpower 
management plan that reflects the complementary roles and responsibilities for the civil 
service and contractor work forces. This plan should ensure that the corporate knowledge 
base is retained within NASA and that it is effectively shared with the contractor teams as 
required to support design, manufacture, test and operations. Mission directorates and 
center directors should ensure that these capabilities are identified and supported. NASA 
engineers should participate actively from concept development through operations. All 
major contracts should fully document and implement these roles and responsibilities. 

4. Lesson: Human space flight experience is critical 

An experienced human space flight manager and staff can make the difference between 
success and failure. The experience level of the management team directly influences 
cost effectiveness of the program through the error rate in technical decision making. 
Stability of that program management team is important; senior managers should remain 
for four years minimum. 

Discussion: 

As described above, space flight is a critical endeavor that requires the highest possible 
performance of all program elements. The program manager and his top staff are critical 
elements of the program, making thousands of decisions that shape the technical, 
operational, and management performance of the system. These decisions can enhance or 
defeat excellence elsewhere in the program. Therefore it is essential that from the start 
the management team has the extensive, relevant experience in human space flight that 
brings them the judgment to make consistently good decisions. 

For example, an ISS program manager’s inexperience caused him to accept late 
requirements from science users and international partners. The cost and technical 
resource shortfalls generated by these late requirements caused a reduction of core system 
elements. Hardware and software were cut from the prime Station systems to save 
money, electrical power, and other critical resources for the new requirements. In order 
to achieve the predicted cost savings, decisions had to be made quickly, without a 
detailed analysis of their consequences. Although the technical manager who led the 
system reduction activity had experience in launch system design and was supported by 
experienced subsystem personnel, he made critical decisions without integrated system 
assessments or experience in long duration, human-carrying spacecraft. As a result, 
Station is significantly less robust than originally designed, with consequent increases in 
technical and operational risk. A more experienced program management team, 
including the program manager, would have realized the high risk they were accepting, 
and likely made a different set of decisions. 
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Technical management of a system as large and complex as a human space system is a 
proactive process. The managers must have not only the technical experience to make 
decisions brought before them, they must also understand which issues require decisions, 
and what questions and answers are needed to support those decisions. They must be able 
to correctly extrapolate previous human space flight experience, both technical and 
operational, to new development and operations problems. There is no substitute for 
subject matter experience and judgment in the management team. 

This does not mean that human space flight experience is required for all on the 
management team. Rather, it requires that a sufficient portion of the senior team be 
experienced human space managers and that the remainder be supported by staff with the 
requisite skills and experience to ensure issues receive the appropriate level of review and 
discussion. 

The path to developing and maintaining this capability is to identify the capabilities 
wherever they reside and make them part of the team. As described in the second 
paragraph of the Executive Lessons section, NASA successfully accomplished this at the 
beginning of the Apollo program. Accomplishing the same function today is many times 
easier because NASA still has many of the technical capabilities within its ra nk s. A 
cooperative partnership between the programs and centers can identify those resources 
and ensure that they are cultivated to support future work. This cultivation includes 
ensuring that young engineers get the opportunity to work on actual hardware systems 
and participate in testing and program reviews to understand the whole process. 

Recommendation: 

Seek out experienced human space flight personnel and put them in positions of 
responsibility early. Establish a career development path, including appropriate technical 
and management mentoring opportunities, to develop a continuous flow of experienced 
space flight managers. Ensure that development program managers have experience in 
flight operations, and vice versa. 

5. Lesson: Under-funded (buy-in) programs are invariably flawed 

Programs that are significantly under-funded at initiation are invariably flawed and often 
un-executable. Program managers often intentionally accept under funding to create 
acceptance for the program (buy-in), with negative consequences. Contingency planning 
and risk management are required to address cost pressure across the life of the program. 

Discussion: 

Cost pressure is a nonnal part of program management. Technical problems, 
requirements changes, natural disasters, changes in political support, and other factors 
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can significantly affect the budget/cost relationship. A major element of the art of 
program management is the development of program plans, system designs, and 
management tools to accommodate such perturbations. In addition, reserves are critical 
elements of program planning. 

However, when a program begins life with substantially less funding than it honestly 
needs to meet its goals, the probability of success is drastically reduced. As previously 
described, human space systems require a high degree of design optimization and have 
small performance margins to start with, which means changes are difficult to 
accommodate without major perfonnance impact and/or drastic cost increases. A 
program manager who accepts an unworkable budget or is forced to make major changes 
due to cost problems will be tempted to take technical or management shortcuts that can 
result in major increases in program risk. In other words, excessive cost pressure leads 
managers to make bad technical and/or management decisions, invariably incurring 
additional risk. This at least reduces the probability of mission success and can easily 
cascade into additional cost and schedule problems during design and development, 
sometimes leading to program termination. At worst it can cause operations costs to 
become unsustainable and/or lead to mission failure and loss of life. 

Programs should be sized to the resources that have been made available to conduct 
them. Even when the basic mission parameters have been defined externally, there may 
remain opportunities to reduce non-critical elements to permit success with the critical 
elements. In most cases, accurate cost estimates are not available until PDR, so 
commitments to program costs and goals should be re-examined at that time. The 
program manager should commit to keep the program content and budget balanced as 
changes in either are observed 

A 20-25% management reserve is generally considered mandatory in all program budgets 
to have a high confidence in success, as well as margins in critical technical perfonnance 
at program initiation. These margins should be maintained as a percentage of the 
remaining work to go. Whenever this level of margin is violated, program content should 
be prioritized and managed to keep that level of effective reserve available for the 
mandatory core content and performance. 

Budget problems invariably continue through the life of the program, as unexpected costs 
arise from technical development problems, test failures, and even acts of nature, such as 
the ISS losses from earthquakes in California. Therefore it is imperative that the program 
objectives and hardware/software design incorporate a significant set of pre-planned and 
continuously updated descope opportunities to enable the successful accomplishment of 
core program objectives. 

New requirements are also a fact of life, whether due to new customers being drawn to 
the effort or improved understanding of what is required to meet the technical objective. 
The latter is very common when a program is the first of its type, as most NASA 
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programs are. In those cases, the program is blazing a new path to achieve its technical 
goals, and new requirements are to be expected as new questions arise and issues are 
uncovered. 

When new requirements come from the customer, they must not be funded from reserves. 
ISS was severely impacted when the program manager and/or administrator accepted 
new requirements without providing the funding to accommodate them. Reserves should 
be held to assure achievement of baseline requirements. 

Innovative technical and management approaches are often offered with promises of 
major cost savings. The history of the space program shows that these are often not 
realistic and/or do not reflect a practical understanding of their effects. The faster, better 
cheaper approach of the 1990s is a classic case of an innovation that was unsuited to 
many major space systems. 

Much of the concern about cost management of human space systems arises from poor 
cost estimates, particularly early in their life cycles. The program manager must have a 
high quality cost estimate to align program content and budget and to prevent under 
funding, and this estimate should be updated continuously during the program. 

The following graphic was presented in the Defense Sciences Board Report on Defense 
Space in 1992. It illustrates the typical history of cost projections from concept to 
program completion. Note that the lowest cost estimate is provided by the contractor, 
who is encouraged by the system to make an unrealistically low bid in order to win. Also 
note the significant cost increase arising from new requirements levied after contract 
award (time dimension is not to scale). Program managers cannot necessarily control this 
behavior, but they must be aware of the trend and plan to accommodate it. In particular, 
the program manager must be on guard for new customers adding unfunded requirements 
late and to prevent the contractor’s bid being used to establish the program budget in lieu 
of the independent cost estimate. 

Finally, it is essential that the program manager keep the right members of management 
informed of problems and other potential sources of cost increases. It may be necessary 
to provide a “rolling update” of cost estimates for particularly ambitious or novel 
programs. Congress, OMB, and NASA HQ have shown a greater willingness to tolerate 
bad news when it is shared in a timely manner and does not appear as a surprise. 
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SBIRS High Quantitative Framework Cost 
Estimate History, 1996-2002 
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Source: BAH Study: Space Systems Development Growth Analysis Report • MPC: Most Probable Cost 

• EAC: Estimate at Completion 

• POE: Program Office Estimate 


Recommendation: 

Beware acceptance of known under funding (buy-in) or contractor under-bidding as it 
introduces new risk to the program. Establish and maintain a cost risk management 
strategy including descoping options to ensure the program manager always has reserves 
(cost, schedule, and technical perfonnance) sufficient to ensure critical program 
objectives can be met in the presence of budgetary pressure. Update cost estimates at 
PDR and only then make commitment on cost. Keep management informed of changes 
before they become surprises. Only accept new requirements that can be independently 
funded. 

6. Lesson: Early identification of appropriate, validated requirements is 
key 
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Early, accurate, and complete requirements definition is critical to program success. 

Effort spent up front to develop appropriate, validated requirements will dramatically 
reduce program cost and improve the probability of successful program execution. 

Discussion: 

Program success depends on the degree to which a program has captured a 
comprehensive and achievable set of requirements at implementation. A lack of 
baselined, technically appropriate, and validated requirements early initiates a series of 
cascading problems that may be unrecoverable. 

The first impact of poor requirements is inefficiency, as a great deal of effort can be lost 
while people pursue inappropriate technical and managerial paths. Good requirements 
communicate critical infonnation about a program including its goals, priorities, technical 
parameters, and management. The absence of good requirements leaves those tenns 
undefined, and all subsequent work is at increased risk of not being aligned thus requiring 
rework later. 

Even when proper effort is applied to requirements definition, pressures to get a rapid 
start often lead to premature system design work in order to show tangible progress. In 
some cases, candidate or strawman requirements are substituted while the real 
requirements are worked out. This usually results in wasted effort and rework with a 
detrimental effect on cost and schedule. 

Engineers and managers who are inexperienced in writing requirements can 
unintentionally either generate or accept poor quality products. This is inexcusable. 
Numerous resources are available to assist program personnel in building and evaluating 
good requirements. The investment in building good requirements up front pays 
generous dividends by shortening the requirements writing phase, improving the 
understanding and cohesiveness of the program team, and improving quality of technical 
and programmatic direction. Also, good requirements can avoid a large number of 
change requests, providing large budget and schedule savings. One study of software 
development programs has shown that requirements growth of 30% during development 
has the potential of doubling the cost of the program. 

Frequently space programs identify too many critical requirements. Critical requirements 
are those whose full implementations are mandatory for program success. Sometimes 
designers attempt to build customer satisfaction by labeling the customer’s requirements 
as critical. While a limited number of critical requirements can be accommodated and 
may be beneficial in clarifying program goals, identifying too many requirements as 
critical can artificially limit the trade space. This can leave the program without a path to 
a successful design implementation. 


23 



Requirements validation is a two-step process. Step one involves tracing the 
requirements’ pedigrees to ensure they all have an owner and are important to the success 
of the program. The second step is equally important but often overlooked - to ensure 
that there is an achievable design solution that will meet the requirement within the 
boundaries of other requirements, including budget and schedule. If no design solution 
can be envisioned within the technical, budget, and schedule trade space, the requirement 
is invalid and should not be accepted. 

For a program such as Exploration, it is important to decide up front what level of 
requirements linkage is to be expected between the development spirals, and ensure it is 
fully incorporated in the baseline. This includes defining and funding technology 
development elements of the program to ensure candidate technologies will have 
achieved the appropriate technology readiness level (TRL) when needed. 

The solution to most requirements management problems is the same. A high quality 
systems-engineering effort will ensure that all requirements are validated up front, are 
well written, meet the intent of the program customers, and can be achieved within the 
cost, schedule, and laws of physics. This is difficult to achieve in practice, and good 
systems engineers are a critical resource, but there is no alternative. 

Recommendation : 

Invest in the best systems engineering and requirements definition capability possible, as 
early as possible, and utilize a robust set of reviews and other techniques to ensure that 
the requirements are adequate, appropriate, and validated for the mission. Limit critical 
requirements to the minimum set absolutely necessary. 

7. Lesson: Plan for the impact of failures 

Space flight is a high risk, low frequency activity. Flight failures, particularly with loss 
of human life, are subject to a great deal of scrutiny and usually result in long periods of 
down time while causes are determined and corrective actions implemented. A long-term 
program of human exploration must account for this in the programmatic approach and 
design of the flight systems. 

Discussion: 

NASA has had three significant accidents with human spacecraft: the Apollo 1 pad fire, 
the loss of Challenger, and the loss of Columbia. In each case, the flight system that was 
the subject of the accident was grounded for a long period of time while the accidents 
were investigated, recommendations were made and implemented, and design issues 
were addressed. The consequences of the grounding were different, with varying impacts 


24 



to customers and other national assets. Future programs need to consider and plan for the 
consequences of reasonable scenarios of vehicle losses. 

In the case of the Apollo 1 fire, the design of the human carrying portion inadequately 
addressed the hazard that resulted in the accident. As a result, the design’s crew 
protection systems were enhanced, and the net effect of the accident was a much more 
robust flight system. In the words of Dr. Chris Kraft, “We would not have made it to the 
moon without that fire.” Even so, there was a significant schedule impact that could have 
been critical if other programs or national assets were depending on it. 

In the case of Challenger and Columbia, the accidents had totally different technical 
failures, but somewhat similar causes in the lack of detailed understanding of the 
performance of the flight systems. In both cases, the entire crew was lost because no 
significant capability had been incorporated to provide crew escape in the flight system 
design. 

The impact to the programs was different each time. Apollo required more than a year to 
address the design deficiencies and make the first flight. Because of the way in which the 
findings of the accident review were embraced by the design teams, the system that flew 
was significantly better than the original design. The program’s success in recovering 
from other system failures on the actual Moon missions reflects the benefits of this 
increased safety and design robustness. 

The Shuttle Program response has been different, and with different consequences. 
Significant improvements followed both accidents as system managers fought to reduce 
residual risk within their own systems. However, due to the enormous impact of 
retrofitting a high performance crew escape system into the flight system, that option has 
not been pursued, and the focus has been to try to avoid any catastrophic failures. As a 
result, a large residual risk remains in flying the Shuttle, resulting in impacts to the users 
of the system. For example, following the Challenger accident, the Department of 
Defense decided to no longer depend on the Shuttle for the launch of critical payloads. 
Likewise, the commercial payload business base was cut off by congressional decree, and 
numerous scientific payloads switched to expendable launchers. 

The consequences of the Columbia failure are even more significant. ISS is the first 
major US space program that is dependent on regular delivery of logistics and crew. 

With the Shuttle grounded, the ISS has been totally dependent on the Russian space 
assets. Were they not part of the equation, the ISS would have been left unattended and 
the Shuttle program would be under great pressure to resume flying as soon as possible to 
prevent the loss of the ISS vehicle. 

Planning for the Exploration program should recognize the consequences of a failure of a 
launch vehicle or other critical system element. A conscious effort should be made to 
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develop an architecture and vehicle designs that are robust to these potential failures and 
which protect the long-term system investment against individual failures. 

For example, system architectures that commit crew survival to successful flight of 
subsequent systems should be avoided as much as possible. Instead, supporting flights 
should be launched ahead of the crew, with sufficient time to ensure that the needed 
capabilities arrive at the point of use in satisfactory condition. 


Recommendation: 

A robust architecture and strategy should be in place to address program responses to 
failures of launch vehicles and other flight elements. This might be assured in part by 
producing an operational Failure Modes and Effects Analysis to assess the operational 
consequences of a major system element. To the extent practical, the crew systems 
should be designed to safely recover the crew 
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4.0 Conclusion 


NASA has a long and distinguished history of accomplishment in human space. From Mercury 
and through ISS, the efforts have been highly successful when technical excellence has been the 
goal and management excellence the route. 

There have been tragic accidents along the way that have taken close friends and colleagues, and 
which have been the source of renewed commitment to excellence in our work. It is important 
that the errors that led to the accidents be recognized, the causes remedied, the lessons they offer 
incorporated into our processes and attitudes. But it is equally important that the soundness of 
the underlying approach be recognized as well. While a small number of tragic errors slipped 
through the nets of engineering and management quality processes, many thousands of 
outstanding decisions have been produced using those processes. We need to strengthen what 
works and fix what is broken, then move on. 

Perhaps the most critical lesson to learn from these accidents, and which is described in 
Executive Lesson #1, is that flying humans in space is extremely difficult, and that the 
technology to do so is less mature than that for most other major technological undertakings. 

The demanding mission of high perfonnance aircraft design and operation is helped by the 
enonnous database of experience that has clearly documented the environments, bounded 
performance uncertainties, and tested our design tools and standards. The unforgiving mission 
of nuclear submarines is assisted by the ability to utilize safety margins far greater than those 
available for spacecraft. Human space has none of those advantages, and the difficulty, expense, 
and probability of loss all reflect it. 

The overriding message to come from our extensive review of lessons learned is this: Flying 
humans in space is hard, and the only way to do it with any degree of assurance is to build on 
and rigorously implement the processes that have worked for the past 40 years. The keys to 
these processes are: 

• Build the team. For Apollo, NASA had to start almost from scratch and develop the 
technical and management teams to do the work. This was greatly facilitated by the 
nearly unlimited funding that was made available to them. But they were extremely 
successful, pulling together an unprecedented technical, manufacturing, and management 
capability have enabled the success of that audacious effort. The core of that team still 
exists, although suffering from neglect and the effects of decades of attrition. The design 
and manufacturing portions are weakest, reflecting the rarity of new human space flight 
systems, but the government core of technical and management capability remains 
because it has been required to support the flight programs with sustaining engineering 
and operations workforce. The Exploration Directorate should develop a plan to sustain 
and rebuild the total workforce required for their mission, including a long-term 
commitment to the NASA workforce that has committed their lives to this endeavor and 
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which constitutes the backbone of the corporate knowledge base on which future work 
will depend. 

• Define the Mission. The Exploration Directorate must develop a mission statement that 
can be easily and consistently articulated to communicate the vision and goals of the 
activity. This is required to accommodate the needs of three principle groups. The first 
group is the public and congress that must support and fund the activity. The message 
they need is the least sophisticated technically, but it must be suited to a lay audience. 

The second group is the team that is working on the program efforts, and they need a 
mission statement that keeps them all focused and working together on the same goals 
and objectives. This is a more detailed technical product, but still generally at the 
conceptual level. The third audience for the message is the team that is going to build the 
requirements and technology plan for the effort. This group needs a much more detailed 
mission statement that gives them the information they need to develop these even more 
detailed system requirements and technology development plans. The Exploration 
Directorate needs to be tightly focused on this activity at this point in its life cycle. 

• Do the systems engineering to determine the requirements and technology 
development planning. This huge job must certainly be near the top of the priority list 
for the Exploration Directorate. In the early days of Apollo, it was almost all-consuming 
for those who needed to lay out the path ahead for the contractor teams and others who 
were to build detailed plans and specifications, and ultimately to build hardware. The 
magnitude of the task was almost overwhelming because the very business of putting 
humans in space was brand new, with virtually no track record or cadre of experienced 
personnel to assist. The circumstances today are totally different. Some groups have 
spent decades examining potential approaches to putting humans on Mars. Other groups 
have accumulated enonnous experience in the design, manufacture, test, and operation of 
the critical systems that will be required to accomplish the mission. These people, many 
of whom are NASA subsystem managers and engineers at the Centers, are eager and able 
to rapidly narrow the trade space for design studies, and can save years of development 
schedule and billions of dollars in the process. This work, if performed in a formal 
process of trade and decision trees, can provide the rigor and traceability to ensure the 
right decisions are made at the right time. 

• Build the bottoms up cost. The long tenn viability of this effort will be enormously 
influenced by the quality of the cost estimates that are generated and the perfonnance of 
the program against those estimates. These estimates will only begin to accurately 
describe the real costs when the systems engineering process allows the design team to 
turn the requirements into design concepts with PDR level fidelity. These cost estimates 
will then be bottoms up, reflecting a realistic understanding of the system design. 

• Adjust the program schedule to fit the funding profile and iterate. Once the 
projected cost and funding rate are known, the schedule can be defined. This is the first 
big test for the program manager, who must try to find a balance between the desires of 
the customer for science return, the engineers for system quality and robustness, and the 
public and congress for rapid progress. The temptation to sacrifice design quality or 
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management rigor in exchange for additional risk is strong, and only exceptional 
judgment and discipline can provide the answer. 

The end result of this review of lessons learned for Exploration can thus be reduced to one 
central theme: human space is one of the most difficult and dangerous pursuits ever undertaken 
by humans and only the very best efforts will be successful. Learn from what has worked in the 
past and build on it. Demand the highest quality in every aspect of the work. Do not cut corners 
or experiment with unproven techniques. Build and support a broad team of technical experts. 

Santayana was correct: “Those who cannot remember the past are condemned to repeat it.” But 
the corollary is equally true: Those who learn from the past can accomplish more than those 
who preceded them. This is the meaning of a learning organization. Incorporating documented 
lessons learned such as these is one critical step on the road to that goal. 
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Appendix A: Selected Lessons 


The following lessons learned have been collected from the various source materials identified 
for our Lessons Learned Study and are representative of the messages found therein. They have 
been grouped by topic to give a broad flavor of the various observations contained in the 
database. In addition, a summary has been provided for each section to characterize the more 
important conclusions to be drawn from them. 

Consideration of these lessons is strongly encouraged for the Exploration Systems Directorate 
management team. However, due to the situation-specific nature of some of the lessons, 
managers are encouraged to review other lessons in the database and/or discuss the subject with 
experts in the topic area and relevant programs/projects before making major decisions. 

A key to the abbreviations denoting the sources can be found at the end of the section. 


Program Management 

General 

Summary: Space programs demand the best management systems and personnel that can be 
found. The tight technical and cost/schedule perfonnance boundaries make success extremely 
demanding. The attention these programs draw from political and media forces makes the job 
even more difficult. New requirements, or politically driven demands, can constrain the program 
to the point where no solution is possible. Demand funding to accompany new requirements, 
and retain reserves to meet the baseline program objectives. 

Over the years, numerous innovations have been offered to make management more efficient. 
Many of these have turned out to be hindrances to the perfonnance of the programs. Extreme 
care should be exercised before implementing a new management approach in a major space 
program. 

1 . The extremely high technical performance and integration requirements of human 
space flight put great stress on the management system. Therefore, the highest 
possible quality of management is required to accomplish the mission with minimum 
cost and schedule. (DSB) 

2. Programs should carefully assess the impacts and relevancy of legacy projects toward 
program goals and objectives before committing resources. (NGLT) 
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3. If it is to be sustainable, the exploration program must be shaped to deliver value to 
America. This will require deliberate management structure and processes to (1) 
understand value, (2) shape to deliver it, and (3) then measure it. Value will ultimately 
shape the technical: mission sequences, crew selection, bandwidth, camera 
configuration, etc. (JSC) 

4. Programs should begin life with a set of requirements and a budget, and a clear 
prioritization of which is superior through its development life. (HQ) 

5. Program reserves should only be used to execute the approved program baseline, not to 
add new requirements. (DSB) 

6. Need someone at the top of the program with the technical ability and authority to 
make the big decisions. Need a program manager with experience equal to the task of 
the program. Recognize that different program management skills, tools and processes 
are required in different phases of a program. What works in Phase B often won’t in 
Phase D. The program manager needs to be the keeper of the vision and objectives for 
the program. Do not rely on process and procedures for this. (JSC) 

7. The engineering technical capabilities of NASA and industry have been declining for a 
long time. A long range program will be required to rebuild and retain the 
engineering/technical capabilities required for this mission. Be prepared to support it. 
(SSC) 

8. To properly control program life cycle costs, the program should assign an operations 
manager at the same level as the vehicle systems manager. This manager is 
responsible for all elements of operations concept development, operations risk 
management, and operations cost management. (MSFC) 

9. Attention to the organization’s discipline, structure, and development is vital to the 
success of the mission. Opportunities to improve these at all levels should be 
encouraged. (NGLT) 

10. The industrial base for human space flight vehicles is not as large, as knowledgeable, 
and as engaged as we had assumed. (NGLT) 
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1 1 . Exploration Systems Directorate management must understand that human space flight 
systems are highly dependent on heritage of design, and that changes do not happen 
quickly. It took ten years to change the computers on the Shuttle. This will have 
significant impact on the spiral development process compared to its application to 
aircraft systems. (MSFC) 

12. Make sure you acquire all the data as well as a piece of hardware. ISS got bitten with 
hardware and not enough technical drawings and specifications to properly sustain the 
hardware. Buying such data after the fact can be costly. Get the "as-built" drawings as 
a contract deliverable! (JSC) 

13. Science is a component of the value the program will deliver to America. It must be 
built into the program's value management system, not tacked on at the margins after 
the "technical" structure is set. Science must be made relevant to the public, media and 
elected officials. It is key to remember the "what's in it for me" mentality of our 
constituencies. (JSC) 

14. Policies and processes that contributed to the success of one mission/project may not 
be appropriate for another. Rigid policies are those that are most easily broken, while 
those that are agile, yet bounded, are most often honored and adhered to. More 
importantly, rigid policies and excessive procedures do not make up for the 
inadequacies in human or infrastructure resources. (ARC) 

15. The frequent changes to both mission/project requirements along with an uncertain 
yearly funding levels are creating an environment where inefficiency and 
ineffectiveness thrives. While mission/project scope and funding changes are aimed at 
reducing risk, the volatility they introduce into mission/project plans actually increases 
risk as transitions from one profile to another opens up new gaps for hazards to enter. 
(ARC) 

16. Stop chasing dreams. In the past decade, $6-7 billion has been wasted on phantom 
programs that were canceled because they were not achievable. Select a rational, 
achievable goal and stick with it. (SSC) 


Cost/Budget 

Summary: Despite the emphasis placed on technical excellence and mission success, the lessons 
clearly indicate an appreciation of the need for constant vigilance toward cost and budget 
performance. The greatest source of problems seems to be twofold: the inability to develop high 


33 



quality cost estimates on which to build dependable program budgets; and the large amount of 
change traffic that poorly planned programs generate and which lead to accelerating cost growth. 


Recommendation: Emphasize the need for a solid plan, including resource weighted schedules, 
before turning on significant design activity. Retain the absolute best cost estimating capability 
for budget planning. 


17. Schedule or cost-driven compromises to the rigorous process of requirements 
development, milestone reviews, and other traditional program management processes 
cause consequences greater than the apparent savings that drove them. (OSP) 

18. Place emphasis on providing credible cost and schedule estimations to stakeholders. 
Take the time to create a NASA team of smart buyers. (OSP) 

19. Contractor cost proposals should NOT be used to validate government budgets unless 
major changes are made to the source selection process to ensure contractor cost 
estimate credibility. (DSB) 

20. Effective life cycle planning requires full understanding of the operations cost 
elements. We have a poor history of attempts to control future operations cost through 
vehicle design because most human vehicles operate so close to the performance 
margins that by the time the design is fixed, operational cost-beneficial elements have 
been scrubbed to solve vehicle performance or cost problems. At the beginning of the 
program, make a conscious effort to identify and trade the DDT&E, Sustaining, and 
Operations Costs. Too often the long-term maintainability and operations efforts 
become a burden of the end-to-end system. Efficiencies are lost through the piece part 
design cuts that occurred over many years. (JSC) 


Planning 

Summary: The quality of the initial planning is a major detenninant of the performance of the 
program. Once the team becomes engaged with the technical and budgetary details of 
implementation, it becomes extremely difficult to find the time for a major reassessment of the 
plan. Centralization of resource planning is essential to ensure that critical resources are 
appropriately applied during peak periods, and to ensure that resources are used in a cost- 
efficient manner. 
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Recommendation: Ensure adequate planning is in place prior to program implementation. 
Solicit independent review of program readiness to proceed. Do not proceed until major plans 
are complete. 


2 1 . The more complex and ambitious the program, the more critical it is to have effective 
planning in place before program implementation or major changes to the program 
plan. A sound basic plan, including schedules and cost estimates codified in a program 
baseline, is the foundation on which the rest of the program’s success is built. The 
decision to accelerate OSP without an implementation plan was a major error. (OSP) 

22. Program management should provide a centralized resource-integration function early; 
the resulting team should have the capability for program- wide budget planning, 
analysis, tracking, reporting, and change control. This is especially critical for 
activities at NASA Centers, where individual project reporting requirements and other 
artifacts of the NASA system can hide or distort cost and schedule issues. A Prime 
Contractor should be help accountable for an integrated summary of all contractor 
performance data. (OSP) 

23. Early identification of long lead items is critical to effective cost and schedule 
management. The development of an integrated program schedule is required to 
identify these long lead items, as dependencies are often not obvious. (OSP) 

24. Long duration programs need to have effective planning in place up front to be able to 
adapt to change in technology, as well as due to obsolescence. Advance planning for 
long-term exploration programs must include plans and resources for establishing 
logistics planning and support programs for fight hardware, test facilities, and mission 
operations support systems. (ASAP) 

25. Initial budget estimates must include budget reserves and schedule margins for each 
major period of performance, with those reserves and margins assigned in proportion 
to the risk in each period. (NGLT) 

26. Ensure that program plans use technical content to drive the schedule, not the other 
way around. (MSFC) 


Roles and Responsibilities 
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Summary: The number three lesson from Section One is to understand and reinforce the roles of 
the government and contractors in the program. The following lessons and others in the backup 
material speak to the details of implementing that lesson. 

Recommendation: Ensure a close but appropriate working relationship between the government 
and contractor management teams. Identify and routinely discuss all issues between the parties, 
including contractual, communications, cultural, and any others that occur. Write down the rules 
of engagement between the parties and operate consistent with them. 


27. Programs should perform a thorough assessment of the manufacturing vendor 
capability required to produce space-qualified hardware for each development spiral. 
(NGLT) 

28. Rules of engagement between the contractors and government must be developed early 
and designed to allow critical work to be done. Concerns and restrictions arising from 
the competitive environment must be identified early and worked into practical 
implementation plans. (OSP) 

29. When a prime contractor is expected to be the user for a technology development 
program, that contractor should be involved throughout the development to assure the 
product has maximum utility with minimum rework. 

30. Program participants should be asked to trade performance elements and 
organizational responsibilities under their control and within their experience base. Do 
not ask contractors to trade Government roles and responsibilities, or the site of 
facilities. Trades should be made to select the best qualified organizations and people, 
not the most convenient. (OSP) 

31. When Centers perfonn work for contractors (e.g., through Government Task 
Agreements), the contractor must be given full responsibility and accountability for 
managing resources. (NGLT) 


Insight/Oversight 

Summary: The differences between insight and oversight may not be commonly understood. 
Regardless of the implementation chosen for Exploration systems, there should be a clear 
understanding by all parties. Common practice would have detailed government oversight of 
high risk activities, where government inspectors and other authorities are in-line to the 


36 



processes. Insight is reserved for the lower risk, turn-the-crank activities where the contractor 
proceeds on their own and keeps the government informed of progress. 


Recommendation: Ensure that appropriate documentation, with thorough supporting rationale, is 
provided for all aspects of government insight/oversight, and that all parties perfonn to the plan. 

32. NASA engineering staffs should be utilized to provide technical insight/independent 
analysis in high risk, mission critical areas. NASA should plan and budget for the 
appropriate level of insight early in the program. Government insight teams and their 
industry counterparts should communicate on a regular basis and continuously look for 
ways to improve implementation methods. Integration is a key building block to an 
effective organization and should be performed on an on-going basis, especially at the 
middle-management level. A lack of integration, cited in the Columbia Accident 
Investigation Board Report as one of the causes of failure, is an issue in NASA’s 
culture. This lack of integration results in turf battles, “stove pipes”, and chains of 
command issuing duplicate and confusing directives. These in turn, adversely impact 
budget, schedule and risk. (ARC) 


33. Every contractor requires a different degree of oversight versus insight. Their Quality 
Assurance (QA) system, even as measured by a standard like ISO, must be proven 
through actual experience, as in-plant inspections can be misleading if contractors put 
on a special show for Government visits. Unfortunately, the degree of required 
oversight/insight has to be detennined empirically such as by the frequency and 
severity of QA issues that arise with the contractor. Projects should have QA plans 
and processes that can be scaled or tailored to variable amounts of insight/oversight 
(based on the Contractor’s performance) such that the amount of resources needed for 
oversight can be reduced and transferred to other vendors who haven’t yet established 
credibility or held in reserve for emergencies. (ARC) 


Communications 

Summary: Communications is the subject of many lessons, and is particularly prominent in 
accident reports. This appears to be an indication of several things. First is the natural tendency 
to avoid discussing bad news, particularly in public. While perfectly natural, this increases the 
likelihood of others repeating the mistake. 

Another concern is the tendency of some individuals and organizations to hoard infonnation. 
This can reflect legitimate concerns for proprietary information, but can seriously hinder the 
engineering and management functions by withholding information critical to perfonning these 
functions. 
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In the worst case, infonnation on critical system shortcomings has failed to make its way to 
decision makers leading to an accident. Both the Columbia and Challenger accidents involved 
such failures of communication. The long and complex communication path from the engineers 
with intimate knowledge of the design details to the top decision makers can make it extremely 
difficult for those at the top to appreciate the significance of all of the inputs they receive. 
Therefore, it is essential that the communications processes of the program be open so that the 
engineers with the concerns feel empowered to come forward and have sufficient dialog with 
senior managers to convey the full context of their concerns. 

Recommendation: Ensure that a process is in place for full and open communications from the 
top of the organization to the working level, and the other way around. The program 
management team should practice management by walking around to emphasize their openness 
to this communication, and to actively solicit inputs from the engineers and technicians. 

34. It is essential that management processes and priorities communicate the correct 
message to employees. The external factors of cost and schedule pressure can have a 
negative influence on safety and reliability. Leadership must ensure that their support 
for other programs and management tools is not allowed to cause “unintended 
consequences” which may force subordinates to make questionable decisions. (ARC) 

35. Education and "public engagement" are components of the value the program will 
deliver to America. They must be built into the program’s value management system, 
not tacked on at the margins after the "technical" structure is set. Public Affairs needs 
to become involved early and stay on board through the life of the Program. They are 
a support function/advisor to the Program and need to help the Program "sell itself' to 
the public. Public Affairs should implement the road show for the Program to the 
public and educational community. Capturing the public's enthusiasm for the mission 
is critical in a program that spans multiple years/political tenures. (JSC, KSC) 


Agreements 

Summary: Inter-agency and inter-governmental agreements are becoming commonplace in 
major programs. Such agreements can offer substantial benefits in terms of enlarging the 
customer base, increasing efficiency by utilizing common infrastructure, and greater political 
support. However, agreements also complicate decision making and can create complex and 
sometimes unworkable management processes. The value and effectiveness of these agreements 
is generally directly proportional to the time and effort invested in them prior to program 
implementation. 

Recommendation: Treat agreements as an element of mandatory planning to be completed prior 
to program implementation. International agreements take a long time to finalize, so begin early 
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and develop a tiered approach to ensure top level agreements are in place before program 
implementation, with an agreed-to schedule for finalizing the lower tier agreements. 


36. Consider International partners early. Minimizing international participation fails to 
capitalize on the benefits of diversity. Diversity is far more than a buzzword 
acknowledging differences. People of differing backgrounds come at decisions and 
arrive at solutions from different perspectives. Assuming different equates to inferior 
is self-defeating as well as the height of arrogance. By openly considering diver 
thoughts we not only expand our options but more fundamentally we are stimulated to 
think beyond our previous boxes. We gain. (HQ) 

37. Inter-Agency coordination of technology development should be pursued at the 
performing level (vs. high level, broad based agreements), capitalizing on hardware 
developed and unique capabilities of the partner agencies. (NGLT) 

38. Partnering agreements should be comprehensive; these should include resources 
(including reserves and procedures for addressing cost growth) and Safety and Mission 
Assurance as well as a clear definition of the roles and responsibilities of each team 
member. (NGLT) 

39. In some cases where industry has lost the capability to develop designs, there may be 
opportunities for cooperative production agreements with industry. (SSC) 


Independent Assessment 

Summary: A well-crafted program of independent assessment can be a program manager’s best 
defense against making dumb mistakes caused either by external pressure or internal myopia. 
Independent cost assessments can warn against excessively optimistic budget estimates, or 
validate good budget requirements and reserve status. Independent technical reviews identify or 
confirm technical and programmatic risk. Independent program reviews can give unbiased input 
on the program’s progress against the plan and identify political and other marginal requirements 
that deserve to be reconsidered. The space business is so challenging, and the political 
environment so all consuming, that independent oversight can be a valuable element of 
management. 

Recommendation: Develop a plan for accommodating independent assessments to assist in 
managing the program. Do not allow fear or ego to obstruct the fact that independent reviews 
can identify legitimate problems that need to be addressed 
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40. Independent assessment is a key ingredient in successful space programs. (HQ) 

4 1 . Independent assessment can greatly facilitate effective management. It can be the 
implementation of the old saying, “Measure twice and cut once.” With IA you are 
measuring twice. (HQ) 

42. In order to avoid the “normalization of deviation”, we need to put major reviews into 
existing programs. A thorough review of requirements, design, and operations every 
3-5 years could catch a problem before an accident. (MSFC) 


Systems Engineering and Requirements Management 


Organization and process: 

Summary: Systems engineering is the process that is used to decompose a large technical project 
into component sections that individual design teams can turn into hardware and software. Later, 
systems engineering assures that these component sections can be integrated into a system that 
satisfies the program-level goals and requirements. This is a critical function for human space 
programs, whose complexity and criticality stress the state of the art in systems engineering. The 
effectiveness of the systems engineering team will be a major indicator of the success of the 
program as a whole. 

The lessons learned database has countless lessons on the value of a good systems engineering 
organization, and the consequences of inadequately supporting this activity. Even when program 
managers intended to have a good systems engineering activity, they have sometimes failed. 
There are expert resources available that can assess the needs of a specific program and mission 
to assist in implementing the correct SE organization. 

Of all of the functions of the Systems Engineering organization, the development of the right 
requirements set is probably the most critical. Space systems have different requirements than 
other undertakings, and human space is the most unique of all. It is essential that the SE 
capability be supplemented with people familiar with the unique needs of the human space flight 
business and that they be given the time and resources to develop a good requirements set during 
formulation. 

Another critical function of the SE organization is oversight of changes to requirements as the 
program proceeds. It is imperative that changes be assessed for all of their impacts to the 
baseline before they are approved for implementation. This should be given preference to 
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having a cost estimate for implementation, as the cost estimate will not be valid without it, and 
the impact of a bad change could involve impacts more serious than cost. 


Recommendation: Assemble the highest quality, most experienced, systems engineering team 
available. Solicit the help of external organization to ensure that the correct capability is in place 
for your specific needs. Empower the team, and give them the resources and authority to do an 
outstanding job. 


43. Thorough and rigorous documentation of the rationale behind all program 
requirements, especially high level requirements defined in the early stages of a 
program, will decrease the expenditure of resources and ensure consistent 
programmatic direction and flowdown of requirements. (JSC) 

44. To develop a complete set of requirements, they should be written as part of a 
functional flow diagram. The use of a functional flow diagram for the level of 
requirements under development can greatly assist in the requirements identification 
and definition. (MSFC) 

45. The Integrated Product Team (IPT) and Analysis and Integration Team (AIT) approach 
served the OSP program well. If utilized, it is essential that all organizations be 
appropriately represented in these teams. (MSFC) 

46. The impact of converting from English to Metric units later, or to carry both 
conventions, is greater than expected. The potential for error in converting between 
units is real. The Exploration programs should plan to utilize metric units from the 
start. (MSFC) 

47. Configuration management is a critical part of a space flight program. This is even 
truer for human space than for other aerospace programs due to the extremely high 
level of system reliability, integration, design optimization, and cost criticality 
required. The many sources of detailed technical data will have varying levels of 
maturity and requirements, and an effective process must be developed to permit the 
flexibility to meet those differing requirements while supporting critical design 
integration activities. A process must be developed for wide availability of approved 
integration data, including strawman or working data available prior to the release of 
baseline data. Configuration management should include analytical tools and models 
as well as data sets, and ensure a consistent set of tools is used across the analytical 
cycle. (JSC) 
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Design Philosophy 

Summary: It is essential that a major program be guided by a design philosophy that supports 
the program goals and is compatible with its resources. Apollo was faced with an enormous 
challenge to develop numerous technologies that were as yet not even conceived. The program 
was successful because it built on a philosophy of large design margins, extensive test, and as 
many contingency plans and fallbacks as could be conceived. Shuttle has suffered as a result of 
having overly optimistic performance expectations from the reliability of some system 
components and from having under-estimated the amount of work required to maintain the 
systems in flight-worthy condition. 

Recommendation: The Exploration Systems Directorate should develop a clear, realistic, and 
technically sound design philosophy that can support the mission within the constraints of 
achievable technology. 


48. The key to minimizing cost growth and operational constraints is large design and 
operational margins. That was the basic philosophical difference between the Saturn 
and Shuttle Programs. With minimal margins, every change becomes a performance 
issue, at the expense of cost, safety, and reliability. With adequate margins, changes 
can be made for cost effectiveness. There should be early trade studies of each 
requirement to determine the cost, performance, safety, reliability, and operational 
sensitivities as an aid in establishing margins, and as a basis for future changes. 
(MSFC) 


49. In order for the systems engineering approach to function correctly, it is essential that 
the “system” be correctly identified as early as possible. For example, the system for a 
human launch system must include all elements of the launch vehicle and its 
supporting systems. The reliability and other operational characteristics of the launch 
system establish numerous critical performance requirements for the crew carrying 
element. At a minimum, they define the crew escape capabilities required of the 
crewed element, but other requirements include structural elements, thennal protection, 
interfaces, etc. Also, the ground support infrastructure affects the crew access and 
emergency egress, abort capabilities, data available to support critical decision making, 
range safety requirements, etc. (FaRC) 

50. A rigorous requirements development process, staffed by personnel familiar with the 
unique characteristics of a human space flight system, is critical to establishing a good 
requirements foundation for program work. A structured approach to requirements 
definition must be followed in a disciplined manner through the design life cycle. The 
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steps in the process are to define: 1) Goals, Needs, Objectives; 2) Concept of 
Operations; 3) Trade Studies, in order. In addition, a clear program objective must be 
defined from which all level 0/1 requirements, and mission concepts can flow. If the 
program will have more than one objective, then linkage between objectives must be 
defined. For example, if the ultimate objective is Mars exploration and the moon is to 
only be a test bed on the way to Mars, then that must be clearly stated and understood. 
At each step along the way the "vision" needs to be clearly communicated to and 
shared by the team. Success criteria should also be defined and agreed to early in the 
program. (JSC) 


51. Develop a "Concept of Operations" as a first step in project formulation (in 5-10 pages, 
what are you trying to do and how do the major pieces fit). Requirements process 
needs to be done correctly: Goals, Needs, Objectives; Concept of Operations; Trade 
Studies; Requirements; Iterate. Take your time to get this process right, don’t be in a 
big rush and try and do it out of order! Get the requirements right. Don’t write vague, 
ambiguous requirements, and hope the contractor will be "creative" with their 
solutions. (This didn’t work for Orbital Space Plane). Need a small, technically well 
versed, non HQ centric, "skunk-works" organization to develop the high level 
architectures. JSC 


52. Although one cannot expect to have system requirements fully defined and matured at 
the start of the program, there should be a comprehensive system sensitivity analysis 
done on the selected configuration/technology base to “flag” early those areas and 
interrelationships that have high potential for significant impacts from changing system 
requirements as the design proceeds. This is sometimes done under the banner of Risk 
Assessment, but too frequently it is simply checking a square rather than a serious tool 
to guide program management. 


53. Configuration management is a critical part of a space flight program. This is even 
truer for human space than for other aerospace programs due to the extremely high 
level of system reliability, integration, design optimization, and cost criticality 
required. The many sources of detailed technical data will have varying levels of 
maturity and requirements, and an effective process must be developed to permit the 
flexibility to meet those differing requirements while supporting critical design 
integration activities. A process must be developed for wide availability of approved 
integration data, including strawman or working data available prior to the release of 
baseline data. Configuration management should include analytical tools and models 
as well as data sets, and ensure a consistent set of tools is used across the analytical 
cycle. 
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54. Don’t confuse tomorrow’s dream with today’s reality. NASA allowed the shuttle to 
effectively transition from a research and development system to operational status 
without the technical rigor to justify the change. The Shuttle had been defined as 
“operational” after 4 test flights, despite the fact that a modern aircraft with less 
inherent criticality and system stress is required to accumulate thousands of test flights 
prior to being declared operational. (Diehl) 

55. Focus only on technology development with a high potential for significant payoff. 
Provide adequate lead over the affected designs with this focused technology 
development. There is a misconception that “new” technology almost automatically 
means lower costs/higher reliability. Frequently, existing technology with well- 
characterized and proven applications may be better than yet to be applied technology. 
(JSC) 

56. Development of a minimum set of NASA preferred design standards for human space 
flight systems (flight and ground) will aid design efforts and prevent the requirements 
creep commonly experienced with gross abuse of applicable documents. (OSP) 

57. Design for operability. Programs sometimes shortchange the design elements required 
to enable efficient operations in favor of short range cost savings. This can be very 
shortsighted, and result in significantly greater lifecycle costs. For example, one OSP 
design had the avionics installed behind the heat shield, requiring a major spacecraft 
disassembly to repair or replace a component. If the problem was identified after 
stacking, this would involve many days of work to remove the vehicle, perform the 
maintenance, and reinstall and reverify connections. A design that allowed for on-the- 
pad access to the avionics would offer huge cost savings and improve operational 
responsiveness. (KSC) 

58. Integrated testing is critical. ACRV program considered fielding the system with no 
flight test. Would have put crew in position of remaining on Station in the presence of 
life-threatening conditions in preference to leaving in an untested vehicle. Hubble hurt 
by lack of integrated test, ISS deleted integrated test early and added back. (HQ) 

59. Test what you fly, fly what you test. Ensure that contractor testing is true to this 
philosophy, and ensure that integrated test is perfonned by NASA prior to flight. 

(SSC) 
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60. Rigorous systems engineering and systems analysis processes should be utilized for 
identifying technical risk to accomplish requirements, technical perfonnance metrics 
(TPM), and figures of merit (FOM) in the formulation stage. These should be used 
continuously to check progress against goals and objectives. (NGLT) 

61. Programs should plan for and require adequate configuration control of analytical 
tools, models, and data sets and lock down the tool suite during analysis cycles. 
(NGLT) 

62. Design safety into the systems. Safety cannot be added on or created with 
documentation. Safety engineers need to work side by side with the design group to 
ensure safety is built in. (SSC) 

63. The purpose of automation is to improve mission performance. Lack of automation 
increases the amount of crew time required to perform mission tasks. Poor automation 
implementation can increase crew time required to perform tasks or worse, render 
them unable to make good decisions when unexpected events occur. Good automation 
implementation will result in good crew situational awareness, which enables them to 
aid in diagnosis and mission decisions. This is especially important when (not if) 
unanticipated failure modes and situations occur. Simpler to operate hardware systems 
result in simpler automation and overall operations. (JSC) 

64. Vehicles with crews are designed differently than vehicles without. For any human 
mission element, human centered design should be practiced. This process starts with 
the human's capabilities and limitations rather than adding the human requirements 
after the design is mostly complete. Some of the requirements for the human that must 
be factored in at the start of the program's life cycle include environmental factors such 
as acoustics, air quality, water quality and radiation limits, as well as habitability 
considerations. One must also identify limiting factors, such as crew hours, EVA 
hours, etc. The design must enhance the human's capabilities not just meet minimum 
performance requirements. The longer the mission duration and the greater the reliance 
on autonomous mission operations, the more important it is to follow Human Design 
Centered principles. (JSC) 


Commercial Off The Shelf (COTS) systems: 

Summary: There has been an increasing interest in utilizing commercially available hardware 
and software as portions of space flight systems and their supporting infrastructure. Experience 
has shown that this is a very satisfactory approach for some items, and a major mistake for 
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others. In general, COTS should not be used as part of any critical systems because of the 
generally lower level of engineering and product assurance used in their manufacture and test. In 
those situations where COTS has been applied to flight systems, such as the laptop computers 
utilized as control interfaces on ISS, the cost of modifying and testing the hardware/software to 
meet flight requirements has far exceeded expectations, potentially defeating the reason for 
selecting COTS in the first place. In other cases, such as the CLCS project at JSC, the cost of 
maintaining the commercial software had not been adequately analyzed and drove the project’s 
recurring costs outside the acceptable range. 

Recommendation: Ensure that candidate COTS products are thoroughly analyzed for technical 
deficiencies and life cycle cost implications before levying them on the program. 


65. COTS systems have potential to reduce system costs, but only if all of their 
characteristics are considered before hand and included in the planned application. 
(Standards) 

66. COTS systems that look good on paper may not scale well to NASA needs for 
legitimate reasons. These include sustaining engineering/update cycle/recertification 
costs, scaling effects, dependence on third party services and products. Need to assure 
that a life-cycle cost has been considered correctly. (HQ - CLCS) 


Requirements Creep 

Summary: One of the most significant threats to programs is the process of requirements creep, 
where a lean set of requirements gradually becomes inflated. This is negative for two principle 
reasons: 

First, genuinely new requirements add content that was not part of the program, and which can 
drive cost and schedule because of the totally new work involved. Even if these new 
requirements are funded with new money (which is often not the case), they compete for existing 
system resources and have impacts on the integrated system design that may not be understood. 
The way to control these requirements is through a formal board approval process that ensures 
appropriate technical assessment and funding approval prior to accepting the requirement. 

The second source of new requirements is increasing specification of detailed requirements 
under an existing higher level requirement. The effect on resources and integration is the same 
as that outlined above, but the cause and corrective action are different. In many cases, new 
requirements are a legitimate response to increased design knowledge as the system matures. 

For example, a better understanding of an environment may require that a heater be added to an 
avionics box in order to maintain its temperature within the qualification range during specific 
portions of the mission not previously considered. Just as common are requirements that come 
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from the personal preferences of engineers and managers that cannot be strictly justified. These 
must be considered carefully, because the intuition of experienced people can be an essential 
element of a successful design. At the same time, excessive conservatism is a problem that can 
seriously hurt a program, so the management team must be extremely careful. 

Recommendation: Ensure that the process of requirement development, control, and approval is 
in place and vigilantly guarding against unnecessary new requirements. Listen to the design 
engineers to ensure that essential requirements are considered. 


67. New technical requirements due to improved understanding of the job should be 
covered by the reserves and/or descoping options. There should be an appropriate 
process in place to process these new requirements, including the resources to perform 
the appropriate technical and management reviews of the impact of the changes. 
Accepting “make work” technical requirements before costs are fully definitized is 
sometimes inescapable, but making technical changes before the systems engineering 
change review process is performed can have huge cost and technical impacts. (ARC) 

68. Particular attention should be given to the design, operation, and cost implications of 
“Political Requirements.” It is not likely that such requirements can be avoided, but 
they should be thoroughly understood and documented for future cost implications. If 
the time comes that such requirements are no longer needed, there should be sufficient 
documentation and trade studies of their design and operational implications that one 
can determine if than be backed out of the design. (ARC) 


Risk Management 

Summary: Risk management is a critical element of all major programs and is a requirement of 
the systems engineering approach. The term risk applies to all major aspects of the program 
goals, including technical performance, budget, schedule, and crew safety/mission success. The 
high risk of human space flight programs has already been established, so the need for risk 
management capability should be self-evident. In fact, this is not necessarily the case, with some 
program management personnel being highly resistant to risk management techniques. There is 
a lingering attitude that ensuring that good designs are produced can replace risk management. 
The lessons database is replete with examples of how this attitude has led programs into trouble. 

The issue is complicated by the shortage of qualified risk management personnel. Many 
managers have little experience with formal risk management, and of those who do, many of the 
experiences have been unpleasant. Risk management is often implemented late in a program, 
after it has gotten into trouble, when the options for resolution are difficult and costly. This can 
make the role of risk manager unpopular. 
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The lessons provide a strong endorsement for a pro-active risk management activity in the 
program from its inception. The benefits of understanding the risks involved in the program and 
incorporating compensating design approaches and other features are well documented. 

Recommendation: An effective risk management program should be implemented as early as 
possible. The program manager should be the prime advocate of risk management in the 
organization, and should demand and decisively act on risk information. 


69. Risk management needs to be an effective part of program management. This is easier 
to say than to do, due to a shortage of trained risk managers, a plethora of risk scoring 
tools from which to choose, and because the decisions that arise from an aggressive 
risk management program can be difficult and costly. (OSP) 

70. Government agencies must manage their own high risk space programs. “Civil 
servants are the ultimate risk takers” (only ones authorized to accept risk). (KSC) 

7 1 . Risk management must be more than a mechanical process of tracking risk. It must be 
an integral element in the formulation of program objectives, requirements, and 
management decision-making. The top program risk manager must be the program 
manager, but he/she should make the philosophy and tools mandatory across the 
organization. (OSP) 

72. Know the real cost of a launch or mission failure, and use it to determine the level of 
investment in mission success. Point is standing army during recovery drives costs 
way up. Challenger and Columbia together probably cost $20B+. (LaRC) 

73. A consistent, continuous risk management system should be utilized across the 
program, and clear pass/fail criteria should be developed for risk reduction activities. 
(NGLT) 

74. Define the risk management system to be used by the program at its onset to, 
integrated with partner programs, contractors, in stakeholders' risk programs, staff 
appropriately, and managed by it. (OSP) 

Safety 
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Safety Organization 

Summary: The role and effectiveness of the safety organization within NASA has been a very 
controversial issue for many years, with strong feelings expressed on all sides. Many feel that 
the right place for the safety engineers is working side by side with the rest of the design team. 
Others feel that they should be set aside in an independent role, where they will be less 
influenced by the peer pressures and other environmental effects of working on the program. 

Recommendation: The senior management team needs to ensure that the safety team has a 
highly visible role, the safety workforce is accorded the same access and prestige as other 
members of the program team, and that the program manager is the top advocate of safety. 

75. There is a large communications and respect gap between the safety personnel and the 
line engineering team. Programs should rotate personnel through S&MA to eliminate 
bias and improve teamwork.(KSC) 

76. The value of S&MA is hugely under-appreciated. S&MA needs to be there working 
closely with the design organization. Needs to find what does not work. The “Power 
of Negative Thinking.” Must challenge the system from concept development to 
operations. (HQ) 

77. S&MA focus is ephemeral. After Challenger, S&MA had lots of clout in getting 
consideration on issues. Over time that clout eroded to where they are virtually 
unheard today. The early focus on safety was unsustainable, so the reaction went too 
far. Need to reach a sustainable long tenn balance. (HQ) 

78. In order to achieve program safety goals, you must put strong S&MA leadership in 
place early in the program. NASA tends to put junior safety engineers on the new 
programs and leave the experienced people on the operational programs. This needs to 
be corrected to ensure that the right requirements are put on the new program. ( MSFC) 


Design for safety 

There are numerous lessons that point to issues with the way we design systems to incorporate 
safety into the design. Most of the lessons are too detailed technically to be included in this 
report, but they should be reviewed in detail during the design phase. One of the recurring 
themes is the mistaken notion that safety requirements may not need to be verified in the same 
way that other technical requirements are. There is rooted the fact that much of the safety data, 
such as reliability predictions, is determined by probabilistic phenomena that are difficult to test, 
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However, there are engineering design principles that implement safe design in the hardware, 
such as true redundancy of critical functions, that can and must be verified. In addition, the 
lessons regarding the approach to the design of the Shuttle indicate that there was an 
unwarranted confidence that the technology was mature enough to offer aircraft-like reliability in 
the system. In fact, the overall reliability of the Shuttle system has proven to be on the same 
order as that of other traditional space flight systems. 

Recommendation: Ensure that the safety requirements are treated with equal or higher respect 
and rigor. Make sure that “show me” is a central tenet of the technology and design maturity 
evaluations. 

79. The risk to the crew is a major driver for technical issues, often leading to late band-aid 
fixes to critical problems, with associated cost and schedule growth. Early claims of 
high reliability in hardware and software are frequently not substantiated by detailed 
analyses such as FMEA, hazard analyses, and probabilistic risk analyses. Realistic or 
conservative expectations of system reliability, reflected in appropriately robust system 
design and crew escape capabilities, are net cost savers. (JSC) 

80. The definition of the flight test program required for a human space flight system is 
one of the most difficult and controversial aspects of the work. Due to cost and other 
reasons, there is a strong desire to limit the flight test to the absolute minimum number. 
However, the complexity and failure history of these systems argue for a more 
substantial flight test program. (JSC) 

8 1 . Definition of safety and mission assurance requirements early in the life cycle is 
critical to the success of the program. The proper balance between rule-based and 
performance based requirements is required to optimize the end product by applying 
the appropriate rigor to the design where needed to meet safety and mission success 
goals. (JSC) 

82. Software product assurance is one of the most critical elements of ensuring a 
successful mission. Independent Validation and Verification is one of the most useful 
tools in ensure that product quality, but all elements of the software development and 
test process must participate. (ASAP) 

83. Safety personnel have a different perspective and mindset than design engineers. 
Instead of considering how the system works, the safety engineer focuses on how it 
fails. The two points of view work off of each other to develop a robust and well 
understood system design. (MSFC) 
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Human Rating 


Summary: NASA recently developed a set of human rating requirements for new space flight 
systems. The requirements are very top level and set the bar high in terms of establishing safety 
goals for the human rated systems. The OSP team found that attempting to incorporate this 
relatively immature and top level guidance into a program can be difficult. The true intent of the 
Human Rating Requirements is clear - maximize the probability of crew survival in a very 
hazardous environment - and deserves to be a key focus of the program’s management and 
design activities. 

Recommendation: The program management and design teams should develop an integrated 
concept of implementation of the human rating requirements for their systems and use that to 
guide the overall system design. 

84. Do not underestimate the impact of human rating and a crew survival requirements all 
aspects of its human space flight program. (OSP) 

85. The current Human Rating Requirements and Guidelines for Space Flight Systems are 
yet to be applied to a flight system, and their effectiveness is unproven. This refers 
both to their ability to protect the safety of the crew, and their practicality in being 
implemented in achievable systems. (OSP) 


Key to sources: 


DSB 

OSP 

NGLT 

ASAP 

Standards 

Diehl 

LaRC 

KSC 

GRC 

JSC 

ARC 

DRFC 

GSFC 

SSC 

MSFC 

JPL 


Defense Sciences Board Report, May 2003 
Orbital Space Plane LL Report 
Next Generation Launch Technology LL Report 
Aerospace Safety Advisory Panel Annual Reports 
NASA Technical Standards Database 

Columbia Lessons - Gen. D. Diehl, USAF, Air and Space Power, Summer 2004 

Langley Research Center Workshop 

Kennedy Space Center Workshop 

Glenn Research Center Workshop 

Johnson Space Center Workshop 

Ames Research Center Workshop 

Dryden Research Flight Center Workshop 

Goddard Space Flight Center Workshop 

Stennis Space Center Workshop 

Marshall Space Flight Center Workshop 

Jet Propulsion Center Workshop 
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Appendix B: 


'ac., 


y 


NA 


Centers Submissions from 
Lessons Learned Workshops 


Ames Research Center 

Dryden Flight Research Center 

Glenn Research Center 
(held individual interviews) 

Goddard Space Flight Center 

Headquarters (Draft) 

Jet Propulsion Laboratory (in review) 

Johnson Space Center 

Kennedy Space Center 

Langley Research Center 

Marshall Space Flight Center 

Stennis Space Center 


Full reports will be provided in a CD furnished with this final report. 
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Appendix C. Source Documentation i. Spreadsheet of Databases 


LESSON LEARNED DATABASES and OTHER WEB-BASED RESOURCES 


Resource Title 


LL Web Databases / Resources (NASA) 


Web Address (URLs 


NASA Lessons Learned Information 


http://llis.nasa.gov/ 


NASA APPL Knowledge Sharing 


System r & 

NASA Lessons Learned Knowledge h ^ ://65 . 168 . 55 . 83/ porta l/site/llkndevl/ 

Network 

,. TA „ A , , http://appl.nasa.gov/businessunits/knowledge/overview/index.ht 

NASA APPL Knowledge Sharing r 

NASA Technical Standards Program http://standards.nasa. gov/npts/login.taf?_function=list 
Flights Programs and Projects 

Directorate (FPPD) Lessons Learned http://eol.gsfc.nasa.gov/miscPages/fppd-ll.html 

Database 

JSC Lessons Learned Database http://iss-www.jsc.nasa.gov/ss/issapt/lldb/ 

NASA Office of Logic Design http://klabs.org/DELlessons_leamed/ 

, , T T . http://www.klabs.org/richcontent/Misc Content/AGC And Hist 

Skylab Lessons Learned < , T , _ _ - 

ory/Skylab/Skylab_Lessons.htm 

Chandra X-ray Observatory Lessons I http://klabs.org/DELlessons_leamed/chandra_lessons.htm 

Lessons Learned From the , ,, , 

. http://www.nap.edu/html/ssb html/Clementine/clem-ch2.shtml 

C lementine Mission _ 

NAS A Kennedy Space Centei Best j ltt p.// www bmp Coe .org/bestpractices/intemal/nasak/index .html 
Manutacturmg Practices 


Skylab Lessons Learned 


http ://www. nap . edu/html/ ssb_html/Clementine/clem-ch2 . shtml 


http://www.bmpcoe.org/bestpractices/internal/nasak/index.html 


NASA Marshall - Best 
Manufacturing Practices 
GSFC Earth Observing Systems 
Lessons Learned 

Shuttle-MIR Lessons learned 

Shuttle Collaboration Lessons 
learned (Boeing) 

ISS payloads Lessons Learned 

Russian LLs 


http://www.bmpcoe.org/bestpractices/internal/nasam/index.html 


http://eos.gsfc.nasa.gov/eos-ll/pilot.html 

http://hedstest.jsc.nasa.gov/history/shuttle-mir/history/to-h-b- 

lessons.htm 

http://orbiterproduction.cal.boeing.com/team3/slessons.nsf 

http://iss-www.jsc.nasa.gov/ss/issapt/payofc/OZ2/LESSONS.doc 
http://mod.jsc.nasa.gov/dx/Russia/eva/learned.htm 


Discovery Program Lessons Learned http://discovery.larc.nasa.gov/discovery/ 
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President's Commission on Moon, 
Mars and Beyond 
Lessons Learned at JPL 


http://www.moontomars.org/ 

http://www.vibrationandshock.com/artl.htm 


LL Web Databases / Resources (Non-NASA - Military, Industry) 

National Nuclear Safety , 

.... . T . http://lessons-leamed.net/detault2.asp 

Administration Lessons Learned 

DOE Corporate Lessons Learned , .. ... 

„ .. . http://www.tis.eh.doe.gov/ll/hstdb.html 

Collection 

The Society for Effective Lessons . .. ... 

T , „ . T http://www.tis.eh.doe.gov/ll/sells/mdex.html 

Learned Sharing (SELLS) 

Center for Army Lessons Learned . .. .. 

http://call.army.mil/ 

1 UALL ) 

Lawrence Livermore National 

T , T T http://www.llnl.gov/es and 1 

Laboratory Lessons Learned - - 


http://www.llnl.gov/es_and_h/lessons/lessons.shtml 


Some Additional General LL Information Web Sites 


Air War College Gateway to Lessons 


http://www.au.af.mi1/au/awc/awcgate/awc-lesn.htm#writing 


T 1 imp ./ / r r r r * * .mi. n i .11111/ n 11/ n y r v/ n r r vt-mv/ 1 

Learned 

Artemis Project - Lessons Learned , .. . , ^ 

» n r ‘ r a 1 ^ • http://www.asi.org/adb/04/06/crew-lessons-trom-mir-5.html 

from Mir 5 About Crew Operations 

Space Engineering Lessons Learned http://users.erols.com/ee/sysengll.htm 
Defense Logistics Agency's A-76 


Competitive Sourcing Internet 
Library 

JSC Quality Management System 
Navy Lessons Learned System 

(NLLS) 

Los Alamos National Laboratory 


http://www.dla.mil/j-3/a-76/a-76main.html 


http://stic.jsc.nasa.gov/dbase/iso9000/master/cwis.cgi 

http://www.nwdc.navy.mil/NLL/NLL.aspx 


Los Alamos National Laboratory https://weblogin.lanl. gov/?referer=https://ps. Ianl.gov/ps7/ 

Anny Acquisition Lessons Learned http://asc.army.mil/divisions/com/all_oview.cfm 
Anny Safety Lessons Learned https://safety.army.mil/pages/lessonslearned/index.html 


54 





Appendix C. Source Documentation 


ii. Bibliography of Reference Reports 

The following is a list of reference documents that were read or searched to some degree, 
and from which some lesson materials and themes were extracted. The Report does not purport 
to be a complete documentation of all applicable lessons learned; that would be an impossible 
task. Rather, the Report is a collection of Lessons that were judged to be relevant and high 
priority for application in the new exploration missions. Therefore, not every reference will 
necessarily be cited in the body of the report. 

1. Office of the President, Washington, DC; Report of the President’s Commission on 
Implementation of United States Space Exploration Policy, A Journey to Inspire, 
Innovate, and Discover; June 2004, Edward C. “Pete” Aldridge, Jr. Chainnan. 
http://www.nasa.gov/pdf/60736main_M2M_report_small.pdf 

2. National Aeronautics And Space Administration, Washington, DC 20546; Report of the 
Columbia Accident Investigation Board; August 2003, Admiral Harold W. Gehman, Jr., 
Chairman (Report not downloaded because of size.) 
http://www.nasa.gov/columbia/caib/htmEstart.html 

3. National Aeronautics And Space Administration, Washington, DC 20546; Report of the 
Diaz Team; A Renewed Commitment to Excellence - An Assessment of the NASA 
Agency-Wide Applicability of the Columbia Accident Investigation Board Report; 
January 30, 2004, Mr. A1 Diaz, Director, Goddard Space Flight Center and Team Lead. 
http://www.nasa.gov/pdf/55691main_Diaz_020204.pdf 

4. Office of the Under Secretary of Defense For Acquisition, Technology, and Logistics 
Washington, D.C. 20301-3140; Report of the Defense Science Board/Air Force Scientific 
Advisory Board Joint Task Force on Acquisition of National Security Space Programs: 
May 2003, Mr. A. Thomas Young, Chairman. 
http://www.acq.osd.mil/dsb/space.pdf 

5. National Aeronautics and Space Administration, Washington, DC, 20546; Report on 
Project Management in NASA, The Mars Climate Orbiter Mishap Investigation Board, 
March 13, 2000, Arthur G. Stephenson, Director, Marshall Space Flight Center and 
Board Chainnan. 

http://appl.nasa.gov/pdf/4736main_marsclimateorbiter.pdf 

http://klabs.org/richcontenEReports/MCO_MIB_Report.pdf 

6. National Aeronautics And Space Administration, Washington, DC 20546; Aerospace 
Safety Advisory Panel; Reports from various years. (Reports not downloaded) 
http://www.klabs.org/richcontent/Reports/NASA_Reports/asap/index.htm 
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7. Office of the President, Washington, DC; Report of The Presidential Commission on the 
Space Shuttle Challenger Accident, June 6, 1986, Mr. William P. Rogers, Chairman. 

(Only recommendations section is downloaded due to size.) 

1) http://history.nasa.gov/rogersrep/genindex.htm 

2) http://history.nasa.gOv/rogersrep/5 llcover.htm 

8. National Aeronautics and Space Administration, Washington, DC, Office of the Chief 
Engineer; A History of NASA Program / Project Management Policies and Processes, 
September, 2002, C. Howard Robbins, Jr., NASA (Ret) 

9. United States General Accounting Office, Washington, DC, Report to the Subcommittee 
on Space and Aeronautics, Committee on Science, House of Representatives; NASA - 
Better Mechanisms Needed for Sharing Lessons Learned; GAO-02- 195; Mr. Allen Li, 
Director Acquisition and Sourcing Management; January 2002. 
http://www.gao.gov/new.items/d02195.pdf 

10. United States General Accounting Office, Washington, DC, Report to The 
Subcommittee on Space and Aeronautics, Committee on Science, House of 
Representatives, Survey of NASA’s Lessons Learned Process, September 5, 2001, Mr. 
Allen Li, Director, Acquisition and Sourcing Management 
http://www.gao.gov/new.items/dO 1101 5r.pdf 

11. National Aeronautics and Space Administration, Washington, DC, 20546; the Mars 
Program Independent Assessment Team (MPIAT), Summary Report, March 14, 2000, 
My. C. Thomas Young, NASA & Lockheed Martin (Ret), Chainnan. 
http://www.jpl.nasa.gov/marsreports/marsreports.html 

12. Jet Propulsion Laboratory, CA; Report on the Loss of the Mars Polar Lander and Deep 
Space 2 Missions; JPL-D- 18709; March 22, 2000; JPL Special Review Board, Mr. John 
Casani, Chairman. 

http://www.jpl.nasa.gov/marsreports/mpiat_report.pdf 

13. National Aeronautics and Space Administration, Washington, DC, 20546, Office of 
Safety, Reliability, Maintainability and Quality Assurance; Lessons Learned from 
Challenger; Lebruary 1988, Planning Research Corporation. 

http://ocw.mit.edu/NR/rdonlyres/Aeronautics-and-Astronautics/16-891JSpace-Policy- 

SeminarSpring2003/B55FC83D-FEBF-4F25-971A- 

6EE371DF5 13B/0/challengerlessons.pdf (Note: this is all one URL address) 

14. National Aeronautics and Space Administration, Washington, DC, 20546; Space Shuttle 
Independent Assessment Team, Report to the Associate Administrator, Office of Space 
Flight, March 7, 2000; Henry McDonald, Chainnan. 
http://www.hq.nasa.gov/osf/siat.pdf 
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15. Lessons from the Shuttle Independent Assessment, presentation by Dr. Tina L. Panontin, 
Chief Engineer, NASA Ames Research Center, RMC III, September 19, 2002 
http://klabs.org/DEI/lessons_leamed/arc_lessons/rmc_siat.pdf 

16. National Aeronautics and Space Administration, Washington, DC, 20546; Mars Climate 
Orbiter Mishap Investigation Board, Phase I Report, November 10, 1999, Mr. Arthur G. 
Stephenson, Director, Marshall Space Flight Center and Board Chairman. 
http://klabs.org/richcontent/reports/MCO_report.pdf 

17. United States General Accounting Office, Washington, DC; Report GAO-03-1 14; Major 
Management Challenges and Program Risks - NASA, January 2003 
http://www.klabs.org/richcontent/Reports/NASA_Reports/asap/index.htm 

18. United States General Accounting Office, Washington, DC; Report GAO-04-642; NASA 
- Lack of Disciplined Cost-Estimating Processes Hinders Effective Program 
Management, May 2004 

http://www.gao.gov/new.items/d02195.pdf 

19. National Aeronautics and Space Administration, Washington, DC, 20546; Report of the 
Advisory Committee On the Future of the U.S. Space Program, December 17, 1990; 
Norman R. Augustine, Chainnan. 

http://www.hq.nasa.gov/office/pao/History/augustine/racfupl.htm 

20. National Aeronautics and Space Administration, Washington, DC, 20546; Enhancing 
Mission Success - A Framework for the Future, A Report by the NASA Chief Engineer 
and the NASA Integrated Action Team; Mr. W. Brian Keegan, NASA Chief Engineer, 
and Ms. Carolyn S. Griner, Chair, NASA Integrated Action Team, December 21, 2000. 
http://www.hq.nasa.gov/office/pao/History/niat.pdf 

21. National Aeronautics and Space Administration, Washington, DC, 20546; Report of the 
Mars Program Independent Assessment Team (MPIAT), March 14, 2000, Mr. A. Thomas 
Young, Chairman. 

http://www.jpl.nasa.gov/marsreports/mpiat_report.pdf 

22. Intelligent Lessons Learned System, International Journal of Expert Systems Research & 
Applications, Vol. 20, No. 1, 17-34, 2001, R. Weber, D. Ana, Becerra-Fernandez. 

23. “Meeting the Project Management Challenge”, Mr. A. Thomas Young; Keynote Address 
to the First Annual NASA Project Management Conference, University of Maryland 
Conference Center, March 30, 2004. 

http://pmchallenge.gsfc.nasa.gov/Presentations/Tom%20Y oung_Keynote%20Address 1 .pdf 
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24. Paper: The NEAR Discovery Mission: Lessons Learned, R. H. Maurer and A. G. Santo, 
The Johns Hopkins University Applied Physics Laboratory, Laurel, MD, 20723; 
http://www.klabs.org/DEI/References/avionics/small_sat_conference/1996/neardis.pdf 

25. National Aeronautics and Space Administration, Washington, DC, 20546; NASA FBC 
Task Final Report, March 2000, Mr. Tony Spear, Task Leader. 
http://appl.nasa.gov/pdf/47305main_FBCspear.pdf 

26. Masters Thesis; Organizational Team Characteristics That Enable Successful Projects at 
NASA - A Framework for the Future, by Timothy J. Flores, Submitted to the System 
Design and Management Program in Partial Fulfillment of the Rrequirements for the 
Degree of Master of Science in Engineering and Business Management at the 
Massachusetts Institute of Technology, June 2001 
http://appl.nasa.gov/resources/archives/flores.html 

27. National Aeronautics and Space Administration, Washington, DC, 20546; NASA SP- 
287, What Make Apollo a Success; 8 Chapters by Low, Kranz, Tindall, Kraft, et.al. 
(Report not available in PDF format, but can be read at the following url: 
http://history.nasa.gov/SP-287/contents.htm ) 

28. American Institute of Aeronautics & Astronautics; Paper AIAA-2001-4630; Joint 
Government / Industry Space Programs: Lessons Learned and Recommendations; Robert 
Bitten, Norman Lao, and Jeff Muhle, The Aerospace Corporation; AIAA Space 2001 
Conference and Exposition, Albuquerque, NM, August 28-30, 2001. 
http://www.aiaa. org/content.cfm?pageid=406 
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Appendix D: Lessons Learned Task and Process Description 


Mission Statement 

This task is to identify, collect, analyze, and integrate lessons learned from prior human and 
robotic space flight programs that have direct applicability to NASA’s new space exploration 
initiative. 

Scope 

This effort is Phase 1 of a 2-phase approach. Phase 1 focuses on an initial survey of high priority 
lessons from previous programs that can assist the Exploration Systems Mission Directorate with 
their work over the next 2 years. The second phase is being planned by the Office of the Chief 
Engineer to continue to collect and develop lessons applicable to later work. 

Phase 1 considers experience from all relevant industries but is focused on space flight activity 
and human missions, particular in the areas of program management, mission and systems 
requirements definition, and safety and mission assurance. Over 500 lessons have been gathered 
from three general activities: (1) mining existing repositories, (2) workshops held at each Center 
and HQ, and (3) direct solicitation of the NASA community. 

The lessons have been distilled into two special groups. The first is a high priority set of 
Executive Lessons that are supported by numerous lessons in the database as well as discussions 
at the workshops. These Executive Lessons have been reviewed by an executive review panel of 
distinguished NASA alumni. Second is a group of individual Selected Lessons that are of 
slightly less priority, but are representative of the lessons contained in the database. 

Collection from Existing Lessons Learned Repositories 

Over 65 databases, reports and other repositories were searched for lessons relevant to the 
formulation of a major initiative or program. Something over 5000 lessons were reviewed and 
considered in screening them down to approximately 550 for inclusion in the Exploration 
Systems database. Candidate lessons were extracted based on potential relevance to the 
Exploration Systems Mission Directorate near-term work, classified, and entered into the web- 
based Lessons Learned Knowledge Network (LLKN). Once entered into the database, the 
lessons were validated, reviewed for and International Traffic in Arms Regulations (ITAR) when 
required, and prioritized. The LLKN Exploration Systems lessons learned database is available at 
url: http://65.168.55.83/portal/site/codet/ . 

Lessons Learned Workshops 
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A series of workshops was held at each NASA field center and Headquarters for the purpose of 
collecting lessons from experienced active duty and retired personnel throughout NASA. 
Facilitated sessions, small discussion groups, direct interviews and other formats were used at 
the discretion of the Center POCs to draw out experiences and develop lessons learned relevant 
to human and robotic space flight.. The workshop coordination and facilitation was led by Mr. 
Tony Mature, Academy of Program and Project Leadership (APPL), Office of the Chief 
Engineer. Each Center had a Center Point-of-Contact (POC) who led the organization of the 
workshops and collection of existing lessons from Center databases. The Center/HQ POCs were 
integral to the overall effort and made it possible to capture a NASA- wide set of important 
lessons. Results of the workshops were documented, vetted through the Centers’ review 
processes, and contributed to the development of the Executive Lessons and the Selected 
Lessons. Where the quality and completeness warranted, individual lessons from workshops are 
included in the lessons learned database 


Direct Solicitation 

Support for archival and hosting of the Exploration Systems database was provided by the 
Lessons Learned Knowledge Network (LLKN) being developed by the Office of the Chief 
Engineer. The LLKN team developed an input page for use in entering the lessons collected by 
the Task Team. NASA employees and on-site contractors were invited to also submit lessons 
via the LLKN web page for possible inclusion in the Exploration Systems database. The web 
page was publicized and the invitation issued through the usual communication / information 
systems at each Center. These lessons are being reviewed, validated, classified and prioritized 
using the same process described below. It is expected that the submission page will remain 
active through September 2004 to allow ample opportunity for submission of individual lessons. 
Lessons received after the completion of this report will be processed and included in the 
Exploration Systems database hosted by the LLKN. Again, the database can be accessed and 
searched at this url: http://65.168.55.83/portal/site/codet/ . 

Evaluation Process 

All lessons collected by the three methods above were evaluated according to a prescribed 
process. The steps include validation, relevance review, prioritization, and ITAR review. 

(1) Validation. The validation process was employed to ensure that the lessons came from a 
documented or credible source. 

(2) Review for Relevance. Lessons were reviewed for relevance or applicability to the 
Exploration Systems mission. As part of this step, lessons were classified according to eight 
classifications: Program Management, Mission and Systems Requirements Definition, Systems 
Engineering and Analysis, Engineering Design, Manufacturing and Assembly, Integration and 
Testing, Mission Operations and Ground Support Systems, and Safety and Mission Assurance. 

In Phase I, emphasis was given on collecting lessons in three of the eight categories: Program 
Management, Mission and Systems Requirements Definition, and Safety and Mission Assurance. 
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(3) Prioritization. The lessons were prioritized as ‘high’, ‘medium’, and Tow’, where high 
denotes lessons that are applicable to program formulation in the next 0-2 years; medium 
lessons are applicable in the next 2-4 years; and low denotes lessons applicable beyond 4 years. 

(4) ITAR Review. Lessons were reviewed to determine whether the technical data contained in 
the lesson were in accordance with ITAR criteria for acceptable release to the public. Lessons 
collected from public domain websites were exempt from this step. The ITAR review was 
conducted by the Langley Research Center Office of the Chief Information Officer and led by 
Mr. Sam Capino, Center Export Administrator. 
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Appendix E: Selected Citations for the Executive Lessons 


1 Human space is different. 

LLIS # 1057 Radiation Exposure 
LLIS # 0476 Zero-G Skills 

LLKN # 1288 Space activities demand the utmost of everyone in any way 
associated with them. (From the Augustine Commission) 

NASA SP-287 What Made Apollo Work 

LLKN #1616 Continual redrafting of goals means Program vision lacks focus. 
Defense Science Board Report, May 2003 

Columbia Accident Investigation Board (CAIB) Report; August 2003 
Tom Young; Keynote Address to the First Annual NASA Project Management 
Conference, March 30, 2004. 

NASA SP-287, What Make Apollo a Success 


2 Focus on mission success. 

Columbia Accident Investigation Board (CAIB) Report, Section 9.3, August 2003 
Lessons Learned from Challenger, # 17 Schedule & cost pressures, inadequate planning. 
LLIS # 0596 Insufficient Understanding and Agreement on Mission Implementation 
LLKN # 1776 (LLIS 0917) Inadequate Project/Line Interaction Throughout Project 
Life Cycle 

LLIS # 0625 Lewis Mission Failure Investigation; lack of understanding of mission 
Requirements, inadequate engineering & management discipline 
CAIB Report 7.5-1 & 9.1-1; Organization authority and oversight is critical 
LLIS # 1303 Good Level 1 requirements enable focus on mission success. 

LLIS # 1257 Define program goals, and provide measure of progress 
NASA SP-287 What Made Apollo a Success 

LLKN # 1648 Even testing must be designed with a focus on mission success. 

LLKN # 1828 (LLIS 1280) Clear vision is needed to maintain focus 

LLKN #1921 Get Centers involved; assign roles & responsibilities in planning. 

LLKN # 1970 Clearly defined requirements are key to maintaining focus. 


3. Understand and reinforce the roles of government and the contractor. 

LLIS # 0596 Insufficient Understanding and Agreement on Mission Implementation 
LLIS # 092 1 Oversight of Contracted S/W Development 

AIAA Paper 2001-4630 Joint Govt/Industry Space Programs: LLs & Recommendations 
Lessons Learned from Challenger; must have independent, effective SR&QA. 
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LLKN # 1928 (LLIS 1343) Grants are not the proper mechanism for acquisition of 
Critical flight hardware. 

LLKN #1771 Verify core competency of contractors. 

LLKN # 1776 (LLIS 0917) Inadequate Project/Line Interaction Throughout Project 
Life Cycle 

LLKN # 1448 Assess competencies of the Contractor management staff through 
their past experiences. 

LLKN # 1447 Balance in-house and contractor S/W development to sustain long- 
term operations capability. 

LLKN #1815 Consider cultural factors in both contractor and international teams. 

LLKN # 1833 Provide proper level management buffers where there are multiple 
management and customer entities. 

LLKN #1681 Clearly define lines of authority and responsibility between programs, 
center support offices and contractors. 

LLKN # 1949 Projects involving multiple Centers require diplomacy and clear definition 
of Center-to-Center roles and responsibilities. 


4. Human space flight experience is critical. 

CAIB Report, Section 9.3 

LLKN # 1800 (LLIS 1058) Shuttle Program/Operations Workforce Competency 
Lessons Learned from Challenger; early planning for crew safety is critical 
LLIS #1312 Experienced personnel are able to recognize risks 
LLIS # 0585 Continuity of experienced staff is key to mission success 
LLIS # 1304 Experienced flight managers are key to effective project org 
LLIS #’s 1421 & 1418 Essential to train & develop new project managers 
LLIS # 1283 Need competent, experienced project managers 
LLIS # 0969 Core team experience in various areas is key 
LLKN #1810 (LLIS 1 172) NASA needs long-term plan to develop & 
retain core competency 

LLKN #1801 (LLIS 1065); lack of knowledge & experience affects operations. 

LLKN # 1880 Competent, experienced technical staff enables good project execution. 
LLKN # 1807 (LLIS 1135) Workforce downsizing and human resources policies are 
having an adverse impact on Management flight experience. 

LLKN # 1680 (Lessons Learned from Challenger) Some NASA and contractor 
management lack motivation, experience and/or skills to manage the 
program effectively. 

LLKN # 1844 (LLIS 1304) A steering group comprised of senior managers from each 
prime mission organization should be utilized. 

LLKN # 1973 (LLIS 1425) The success of a project is greatly impacted by the 
experience and management style of the Project Manager. 

LLKN # 1289 MGS and Pathfinder success can be directly attributed to the experienced 
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project managers and their effective use of expertise from numerous sources. 
LLKN # 1659 (from ASAP 1990 report) Everything possible should be done to ensure 
technical and managerial continuity of the program. 


5. Under-funded programs are invariably flawed. 

Tom Young Keynote Address to the First Annual NASA Project Management 
Conference, March 30, 2004. 

Mars Program Independent Assessment Team (MPIAT) Report, March 14, 2000 
Columbia Accident Investigation Board (CAIB) Report, Section 9.3, August 2003 
LLIS # 0625 Lewis Mission Failure Investigation; cost and schedule pressure 
LLIS # 0904 ESSP VCL was under-funded. 

LLIS # 0889 Technology missions require higher budgetary reserves 
LLIS # 1344 Credible cost estimating 

Lessons Learned from Challenger; adequate resources and contingency planning 

LLKN # 1260 Provide realistic estimate of needed resources 

LLKN #1818 (LLIS 1231) Insufficient funding for current Shuttle & ISS ops. 

LLKN # 1678(from CAIB Report) Working with an increasingly constrained budgets 
- trying to do too much with too little is detrimental to program success. 

LLKN # 1845 Establishing and managing adequate management reserves is essential. 
LLKN # 1689 (ARC LL database) An initial cost estimate that is significantly inaccurate 
can doom a project to failure regardless of what other actions are taken. 

LLKN # 1929 A project's budget should never be forced to fit into an available funding 
source if the requirements dictate the need for additional resources. 

LLKN # 1430 Characteristics peculiar to technology validation missions will require 
additional reserves. 

LLKN #1831 (LLIS 1283) Project Management must have a degree of technical 
competency to make sound decisions and maintain support from project 
engineers and scientists. 


6. Early identification of appropriate, validated requirements is key. 

LLIS # 0625 Lewis Mission Failure Investigation; requirements changes 
CAIB Report; 7.5-1 requirements management; 7.4-3 risk management 
LLIS # 0229 Validated requirements are needed to establish schedules 
LLIS # 1303 Level 1 requirements must be clearly defined. 

NASA SP-287 What Made Apollo Work 

LLKN # 1970 Clearly defined requirements is essential to plan work and maintain focus. 
LLKN # 1649 (from CAIB report) Definition of Level II/III requirements requires the 
involvement of a wide spectrum of experienced personnel. 

LLKN # 1682 (from MPIAT report) Project managers must assure proper development 
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of requirements and be responsible and accountable for meeting them. 

LLKN # 1750 Early recognition of the importance of contamination-control during every 
phase of development and hardware design will avoid costly changes later. 

LLKN # 1735 Requirements must be clearly defined and flowed down, with clear, 
documented definition of testing / analysis required for verification. 

LLKN #2109 Logistics products and requirements must developed early and in 
sufficient detail to allow for proper planning and development. 


7. Plan for the impact of failures. 

LLIS #1312 A culture that emphasizes risk management is essential 
LLKN # 1802 Plan backups for technologies that do not make the need gates. 
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Appendix F : Recommendation for Phase 2 Lessons Learned 


It is the strong recommendation of this team that the Exploration Directorate maintain an on- 
going commitment to lessons learned activities. The intent of this would be threefold: 

To continue to mine existing lessons learned databases for new information relevant to 
the then-current stage of activities. The present activity was focused on program 
formulation and early requirements development, but later stages will be focused on 
hardware development, test, and operations. 

To update the database with lessons learned by the Exploration team as it tackles this new 
challenge. 

To assess the perfonnance of the team in its efforts to apply the lessons that have been 
presented to it. History clearly shows the tendency to repeat the mistakes of the past, and 
it will require vigilance on the part of the management team to ensure that they are 
effective in breaking that cycle. 

The recommended approach would be to utilize the expertise and manpower of the Office of the 
Chief Engineer to perform the functions of lessons learned management for them. This would 
free the Exploration team from the burden of managing the databases, conducting training, 
performing studies, and assessing perfonnance against the lessons. The Exploration Directorate 
would be the customer for this service and would only need to negotiate the requirements and 
associated costs of the effort. 

Much of this work is being perfonned by JPL, and significant aspects are being performed by 
other contractors. The OCE would provide the management and technical direction/oversight for 
the Exploration Directorate. 

Specific recommended tasks would include: 

Continue to study and mine the lessons learned databases for information appropriate to 
each phase of the program work. Sort the output and provide as appropriate to all 
performing organizations. 

Provide sufficient analysis to enable the Exploration team to quickly appreciate the 
underlying message in the lessons, including the historical precedents that can 
significantly change the significance of individual lessons. 

Study the history of significant program not represented in the lessons learned database, 
particularly foreign programs. The Russian space program is a wealth of infonnation for 
operations and hardware design, but significant study and expertise is required to 
interpret it correctly. Provide summaries of appropriate lessons to the Exploration 
Directorate. 
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ERRATA 


Review of the Appendix C, Source Documentation, Item i, Spreadsheet of Databases, 
indicated that many of the web-based resources no longer provide valid links. Below are 
the resources that take the user to relevant web locations as of June, 2015. Other 
entries in Appendix C-i no longer appear to have valid URLs. 


LESSON LEARNED DATABASES and OTHER WEB-BASED 
RESOURCES 


Resource Title 

Web Address (URLs) 

NASA Lessons Learned 
Information System 

http://llis.nasa.gov/ 

NASA Technical Standards 
Program 

http://standards.nasa.gov 

JSC Lessons Learned Database 

http://iss-www.isc.nasa.gov 

NASA Office of Logic Design 

http ://klabs . org/DEI/lessons learned/ 

Chandra X-ray Observatory 
Lessons Learned 

http://klabs.org/DEI/lessons leamed/chandra lessons.htm 

Lessons Learned From the 
Clementine Mission 

http://www.nap.edu/html/ssb html/Clementine/clem-ch2.shtml 

NASA Marshall - Best 
Manufacturing Practices 

http://www.bmpcoe.org/bestpractices/intemal/nasam/index.html 

Discovery Program Lessons 
Learned 

http://discoverv.larc.nasa.gov/discoverv/ 

President's Commission on 
Moon, Mars and Beyond 

http :// www. moontomars . or g/ 

Air War College Gateway to 
Lessons Learned 

http://www.au.af.mi1/au/awc/awcgate/awc-lesn.htm#writing 

Artemis Project - Lessons 
Learned from Mir 5 About 
Crew Operations 

http://www.asi.org/adb/04/06/crew-lessons-from-mir-5.html 

JSC Quality Management 
System 

https://qmsmasterlist.isc.nasa.gov/Home.aspx/lndex 

Los Alamos National 
Laboratory 

https://weblogin.lanl.gOv/7refererMittps://ps.lanl.gov/ps7 
(requires login) 


